May 20 13:47:51.484828 ip-10-0-129-117 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 20 13:47:51.484839 ip-10-0-129-117 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 20 13:47:51.484849 ip-10-0-129-117 systemd[1]: kubelet.service: Failed with result 'resources'.
May 20 13:47:51.485157 ip-10-0-129-117 systemd[1]: Failed to start Kubernetes Kubelet.
May 20 13:48:01.604687 ip-10-0-129-117 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 20 13:48:01.604707 ip-10-0-129-117 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a01fb190102f444db4d677ead2fec2be --
May 20 13:50:21.751668 ip-10-0-129-117 systemd[1]: Starting Kubernetes Kubelet...
May 20 13:50:22.223674 ip-10-0-129-117 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:22.223674 ip-10-0-129-117 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 20 13:50:22.223674 ip-10-0-129-117 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:22.223674 ip-10-0-129-117 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 20 13:50:22.223674 ip-10-0-129-117 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 13:50:22.225269 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.225169    2550 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 20 13:50:22.227965 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.227948    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:22.227965 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.227965    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.227971    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.227976    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.227979    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.227982    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.227985    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.227988    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.227991    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.227995    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.227998    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228002    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228004    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228007    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228010    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228012    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228015    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228017    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228020    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228022    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:22.228030 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228025    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228028    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228031    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228035    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228038    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228040    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228043    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228046    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228050    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228052    2550 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228055    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228058    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228060    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228063    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228065    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228068    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228071    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228073    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228076    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228078    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:22.228494 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228081    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228083    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228086    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228088    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228091    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228093    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228096    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228099    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228102    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228104    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228107    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228109    2550 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228112    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228114    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228117    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228120    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228123    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228125    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228128    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228130    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:22.228971 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228133    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228135    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228138    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228141    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228143    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228146    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228148    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228152    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228154    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228157    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228160    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228162    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228165    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228167    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228171    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228174    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228176    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228178    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228182    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228185    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:22.229487 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228187    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228190    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228192    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228195    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228197    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228200    2550 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228592    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228598    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228601    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228603    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228606    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228609    2550 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228612    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228615    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228618    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228620    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228623    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228626    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228628    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228631    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:22.229962 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228633    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228636    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228639    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228641    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228644    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228648    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228659    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228662    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228664    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228667    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228670    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228672    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228677    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228680    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228683    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228686    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228689    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228692    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:22.230458 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228695    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228698    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228701    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228704    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228707    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228710    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228712    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228715    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228718    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228720    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228723    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228726    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228728    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228731    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228734    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228736    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228739    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228741    2550 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228744    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228746    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:22.230939 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228750    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228753    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228756    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228759    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228763    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228765    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228768    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228770    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228773    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228775    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228778    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228780    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228783    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228785    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228788    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228791    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228793    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228796    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228798    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228801    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:22.231466 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228803    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228806    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228808    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228810    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228813    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228815    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228818    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228820    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228823    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228825    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228828    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228831    2550 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228833    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.228836    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228926    2550 flags.go:64] FLAG: --address="0.0.0.0"
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228937    2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228947    2550 flags.go:64] FLAG: --anonymous-auth="true"
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228954    2550 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228959    2550 flags.go:64] FLAG: --authentication-token-webhook="false"
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228962    2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228967    2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 20 13:50:22.231950 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228972    2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228975    2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228979    2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228982    2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228985    2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228988    2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228992    2550 flags.go:64] FLAG: --cgroup-root=""
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228995    2550 flags.go:64] FLAG: --cgroups-per-qos="true"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.228998    2550 flags.go:64] FLAG: --client-ca-file=""
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229000    2550 flags.go:64] FLAG: --cloud-config=""
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229003    2550 flags.go:64] FLAG: --cloud-provider="external"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229006    2550 flags.go:64] FLAG: --cluster-dns="[]"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229010    2550 flags.go:64] FLAG: --cluster-domain=""
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229013    2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229016    2550 flags.go:64] FLAG: --config-dir=""
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229019    2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229023    2550 flags.go:64] FLAG: --container-log-max-files="5"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229027    2550 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229033    2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229036    2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229039    2550 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229042    2550 flags.go:64] FLAG: --contention-profiling="false"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229045    2550 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229049    2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229052    2550 flags.go:64] FLAG: --cpu-manager-policy="none"
May 20 13:50:22.232459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229055    2550 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229063    2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229066    2550 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229069    2550 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229072    2550 flags.go:64] FLAG: --enable-load-reader="false"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229075    2550 flags.go:64] FLAG: --enable-server="true"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229077    2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229082    2550 flags.go:64] FLAG: --event-burst="100"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229085    2550 flags.go:64] FLAG: --event-qps="50"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229088    2550 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229091    2550 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229094    2550 flags.go:64] FLAG: --eviction-hard=""
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229098    2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229101    2550 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229104    2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229107    2550 flags.go:64] FLAG: --eviction-soft=""
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229110    2550 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229113    2550 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229116    2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229119    2550 flags.go:64] FLAG: --experimental-mounter-path=""
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229122    2550 flags.go:64] FLAG: --fail-cgroupv1="false"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229125    2550 flags.go:64] FLAG: --fail-swap-on="true"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229127    2550 flags.go:64] FLAG: --feature-gates=""
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229131    2550 flags.go:64] FLAG: --file-check-frequency="20s"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229135    2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 20 13:50:22.233050 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229139    2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229142    2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229145    2550 flags.go:64] FLAG: --healthz-port="10248"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229149    2550 flags.go:64] FLAG: --help="false"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229152    2550 flags.go:64] FLAG: --hostname-override="ip-10-0-129-117.ec2.internal"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229155    2550 flags.go:64] FLAG: --housekeeping-interval="10s"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229158    2550 flags.go:64] FLAG: --http-check-frequency="20s"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229161    2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229164    2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229169    2550 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229172    2550 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229175    2550 flags.go:64] FLAG: --image-service-endpoint=""
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229178    2550 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229181    2550 flags.go:64] FLAG: --kube-api-burst="100"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229183    2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229186    2550 flags.go:64] FLAG: --kube-api-qps="50"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229189    2550 flags.go:64] FLAG: --kube-reserved=""
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229192    2550 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229195    2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229198    2550 flags.go:64] FLAG: --kubelet-cgroups=""
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229201    2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229204    2550 flags.go:64] FLAG: --lock-file=""
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229206    2550 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229209    2550 flags.go:64] FLAG: --log-flush-frequency="5s"
May 20 13:50:22.233676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229213    2550 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229218    2550 flags.go:64] FLAG: --log-json-split-stream="false"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229221    2550 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229224    2550 flags.go:64] FLAG: --log-text-split-stream="false"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229227    2550 flags.go:64] FLAG: --logging-format="text"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229230    2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229233    2550 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229240    2550 flags.go:64] FLAG: --manifest-url=""
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229244    2550 flags.go:64] FLAG: --manifest-url-header=""
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229249    2550 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229252    2550 flags.go:64] FLAG: --max-open-files="1000000"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229256    2550 flags.go:64] FLAG: --max-pods="110"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229259    2550 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229262    2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229265    2550 flags.go:64] FLAG: --memory-manager-policy="None"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229269    2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229272    2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229275    2550 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229278    2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229286    2550 flags.go:64] FLAG: --node-status-max-images="50"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229289    2550 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229292    2550 flags.go:64] FLAG: --oom-score-adj="-999"
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229295    2550 flags.go:64] FLAG: --pod-cidr=""
May 20 13:50:22.234249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229298    2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229304    2550 flags.go:64] FLAG: --pod-manifest-path=""
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229307    2550 flags.go:64] FLAG: --pod-max-pids="-1"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229310    2550 flags.go:64] FLAG: --pods-per-core="0"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229313    2550 flags.go:64] FLAG: --port="10250"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229316    2550 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229319    2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e5e41dfa6465a280"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229322    2550 flags.go:64] FLAG: --qos-reserved=""
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229325    2550 flags.go:64] FLAG: --read-only-port="10255"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229328    2550 flags.go:64] FLAG: --register-node="true"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229331    2550 flags.go:64] FLAG: --register-schedulable="true"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229346    2550 flags.go:64] FLAG: --register-with-taints=""
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229350    2550 flags.go:64] FLAG: --registry-burst="10"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229353    2550 flags.go:64] FLAG: --registry-qps="5"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229356    2550 flags.go:64] FLAG: --reserved-cpus=""
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229359    2550 flags.go:64] FLAG: --reserved-memory=""
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229363    2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229368    2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229371    2550 flags.go:64] FLAG: --rotate-certificates="false"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229374    2550 flags.go:64] FLAG: --rotate-server-certificates="false"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229377    2550 flags.go:64] FLAG: --runonce="false"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229380    2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229383    2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229387    2550 flags.go:64] FLAG: --seccomp-default="false"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229390    2550 flags.go:64] FLAG: --serialize-image-pulls="true"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229393    2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 20 13:50:22.234841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229396    2550 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229399    2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229403    2550 flags.go:64] FLAG: --storage-driver-password="root"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229406    2550 flags.go:64] FLAG: --storage-driver-secure="false"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229409    2550 flags.go:64] FLAG: --storage-driver-table="stats"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229412    2550 flags.go:64] FLAG: --storage-driver-user="root"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229415    2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229418    2550 flags.go:64] FLAG: --sync-frequency="1m0s"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229421    2550 flags.go:64] FLAG: --system-cgroups=""
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229423    2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229430    2550 flags.go:64] FLAG: --system-reserved-cgroup=""
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229432    2550 flags.go:64] FLAG: --tls-cert-file=""
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229435    2550 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229439    2550 flags.go:64] FLAG: --tls-min-version=""
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229442    2550 flags.go:64] FLAG: --tls-private-key-file=""
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229445    2550 flags.go:64] FLAG: --topology-manager-policy="none"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229448    2550 flags.go:64] FLAG: --topology-manager-policy-options=""
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229451    2550 flags.go:64] FLAG: --topology-manager-scope="container"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229454    2550 flags.go:64] FLAG: --v="2"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229458    2550 flags.go:64] FLAG: --version="false"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229462    2550 flags.go:64] FLAG: --vmodule=""
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229466    2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.229470    2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229567    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:22.235545 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229571    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229575    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229578    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229581    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229584    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229586    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229589    2550 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229592    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229595    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229597    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229600    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229603    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229606    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229608    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229611    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229614    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229616    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229620    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229624    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:22.236159 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229627    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229630    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229633    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229636    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229638    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229641    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229644    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229647    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229649    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229652    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229654    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229658    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229663    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229665    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229668    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229671    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229673    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229676    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229679    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229681    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:22.236670 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229684    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229687    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229689    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229692    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229694    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229697    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229700    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229702    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229705    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229707    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229710    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229712    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229715    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229718    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229721    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229725    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229727    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229730    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229732    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229735    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:22.237181 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229738    2550 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229740    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229743    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229745    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229750    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229753    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229755    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229758    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229761    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229763    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229765    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229768    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229770    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229773    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229776    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229778    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229781    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229783    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229786    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229789    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:22.237701 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229792    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229794    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229797    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229799    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229802    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.229804    2550 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.230469    2550 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.237470    2550 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.237486    2550 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237535    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237540    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237543    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237546    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237550    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237552    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237555    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:22.238200 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237558    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237560    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237563    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237566    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237570    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237574    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237577    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237579    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237582    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237585    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237587    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237590    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237592    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237595    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237598    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237600    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237603    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237605    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237608    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237611    2550 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:22.238708 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237613    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237616    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237618    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237621    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237626    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237629    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237631    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237634    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237636    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237639    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237642    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237644    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237647    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237650    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237652    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237655    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237658    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237660    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237663    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237665    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:22.239251 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237668    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237671    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237673    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237676    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237678    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237681    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237685    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237689    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237692    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237695    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237698    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237700    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237703    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237706    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237709    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237712    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237715    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237719    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237721    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:22.239774 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237724    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237727    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237729    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237732    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237734    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237738    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237740    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237743    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237746    2550 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237748    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237751    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237753    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237756    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237758    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237761    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237764    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237767    2550 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237769    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237771    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:22.240236 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237774    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.237779    2550 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237907    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237913    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237916    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237919    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237923    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237925    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237928    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237931    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237936    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237940    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237944    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237947    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237950    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 13:50:22.240735 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237953    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237956    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237958    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237961    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237963    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237966    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237968    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237971    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237974    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237976    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237979    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237981    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237984    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237986    2550 feature_gate.go:328] unrecognized feature gate: Example2
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237989    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237992    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237994    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.237997    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238000    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238003    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 13:50:22.241130 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238005    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238008    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238010    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238013    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238015    2550 feature_gate.go:328] unrecognized feature gate: Example
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238018    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238021    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238024    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238027    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238030    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238038    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238041    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238043    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238063    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238066    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238069    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238072    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238075    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238078    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238080    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 13:50:22.241641 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238083    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238085    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238088    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238090    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238093    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238095    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238098    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238101    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238103    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238106    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238109    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238111    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238114    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238116    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238119    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238121    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238124    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238126    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238129    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238132    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 13:50:22.242239 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238134    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238137    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238139    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238150    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238154    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238157    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238159    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238162    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238164    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238167    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238169    2550 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238172    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:22.238174    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.238179    2550 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.238845    2550 server.go:962] "Client rotation is on, will bootstrap in background"
May 20 13:50:22.242807 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.240784    2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 20 13:50:22.243229 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.241679    2550 server.go:1019] "Starting client certificate rotation"
May 20 13:50:22.243229 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.241780    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 13:50:22.243229 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.242572    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 13:50:22.265197 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.265169    2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 13:50:22.267847 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.267824    2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 13:50:22.283632 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.283608    2550 log.go:25] "Validated CRI v1 runtime API"
May 20 13:50:22.288463 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.288446    2550 log.go:25] "Validated CRI v1 image API"
May 20 13:50:22.289816 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.289799    2550 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 20 13:50:22.293143 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.293125    2550 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 13:50:22.295670 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.295649    2550 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 87e66d49-e3a7-4520-b2ce-9c2613a63de9:/dev/nvme0n1p3 c295e943-9771-4278-b9c3-fe1cf7965f89:/dev/nvme0n1p4]
May 20 13:50:22.295740 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.295668    2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 20 13:50:22.300225 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.300119    2550 manager.go:217] Machine: {Timestamp:2026-05-20 13:50:22.299602337 +0000 UTC m=+0.425985481 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091796 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d4d6f4a1474ade2a003c4b22e6b02 SystemUUID:ec2d4d6f-4a14-74ad-e2a0-03c4b22e6b02 BootID:a01fb190-102f-444d-b4d6-77ead2fec2be Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5d:24:5a:b4:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5d:24:5a:b4:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:f5:16:88:27:6f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 20 13:50:22.300825 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.300815    2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 20 13:50:22.300908 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.300896    2550 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 20 13:50:22.302923 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.302900    2550 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 20 13:50:22.303106 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.302925    2550 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 20 13:50:22.303152 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.303119    2550 topology_manager.go:138] "Creating topology manager with none policy"
May 20 13:50:22.303152 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.303128    2550 container_manager_linux.go:306] "Creating device plugin manager"
May 20 13:50:22.303152 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.303135    2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 13:50:22.303606 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.303596    2550 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 13:50:22.305277 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.305267    2550 state_mem.go:36] "Initialized new in-memory state store"
May 20 13:50:22.305566 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.305556    2550 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 20 13:50:22.307963 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.307953    2550 kubelet.go:491] "Attempting to sync node with API server"
May 20 13:50:22.307996 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.307968    2550 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 20 13:50:22.307996 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.307980    2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 20 13:50:22.307996 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.307989    2550 kubelet.go:397] "Adding apiserver pod source"
May 20 13:50:22.308096 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.307998    2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 20 13:50:22.309102 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.309090    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 13:50:22.309155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.309109    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 13:50:22.309904 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.309885    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qfm2q"
May 20 13:50:22.312322 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.312302    2550 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 20 13:50:22.314025 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.314009    2550 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 20 13:50:22.315760 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315740    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 20 13:50:22.315837 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315767    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 20 13:50:22.315837 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315778    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 20 13:50:22.315837 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315787    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 20 13:50:22.315837 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315800    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 20 13:50:22.315837 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315811    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 20 13:50:22.315837 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315821    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 20 13:50:22.315837 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315836    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 20 13:50:22.316133 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315857    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 20 13:50:22.316133 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315867    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 20 13:50:22.316133 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315881    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 20 13:50:22.316133 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.315890    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 20 13:50:22.318371 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.318355    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 20 13:50:22.318447 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.318435    2550 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 20 13:50:22.318673 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.318651    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qfm2q"
May 20 13:50:22.319785 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.319753    2550 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 20 13:50:22.320154 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.320131    2550 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 20 13:50:22.321750 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.321733    2550 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 20 13:50:22.322073 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.322057    2550 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 20 13:50:22.322141 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.322104    2550 server.go:1295] "Started kubelet"
May 20 13:50:22.322190 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.322141    2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 20 13:50:22.322304 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.322259    2550 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 20 13:50:22.322367 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.322320    2550 server_v1.go:47] "podresources" method="list" useActivePods=true
May 20 13:50:22.322984 ip-10-0-129-117 systemd[1]: Started Kubernetes Kubelet.
May 20 13:50:22.324010 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.323896    2550 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 20 13:50:22.324741 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.324723    2550 server.go:317] "Adding debug handlers to kubelet server"
May 20 13:50:22.332494 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.332470    2550 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 20 13:50:22.334645 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.334614    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 20 13:50:22.335099 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.335084    2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 20 13:50:22.335390 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.335374    2550 volume_manager.go:295] "The desired_state_of_world populator starts"
May 20 13:50:22.335390 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.335388    2550 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 20 13:50:22.335517 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.335416    2550 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 20 13:50:22.335517 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.335509    2550 reconstruct.go:97] "Volume reconstruction finished"
May 20 13:50:22.335589 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.335522    2550 reconciler.go:26] "Reconciler: start to sync state"
May 20 13:50:22.335807 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.335784    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:22.336188 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.336171    2550 factory.go:55] Registering systemd factory
May 20 13:50:22.336265 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.336212    2550 factory.go:223] Registration of the systemd container factory successfully
May 20 13:50:22.336428 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.336417    2550 factory.go:153] Registering CRI-O factory
May 20 13:50:22.336428 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.336429    2550 factory.go:223] Registration of the crio container factory successfully
May 20 13:50:22.336508 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.336470    2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 20 13:50:22.336508 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.336483    2550 factory.go:103] Registering Raw factory
May 20 13:50:22.336508 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.336497    2550 manager.go:1196] Started watching for new ooms in manager
May 20 13:50:22.336767 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.336758    2550 manager.go:319] Starting recovery of all containers
May 20 13:50:22.338507 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.338484    2550 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:22.340903 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.340746    2550 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-117.ec2.internal\" not found" node="ip-10-0-129-117.ec2.internal"
May 20 13:50:22.346706 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.346687    2550 manager.go:324] Recovery completed
May 20 13:50:22.352032 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.352012    2550 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:22.353991 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.353970    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:22.354065 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.354011    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:22.354065 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.354027    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:22.354543 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.354527    2550 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 20 13:50:22.354543 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.354541    2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 20 13:50:22.354651 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.354559    2550 state_mem.go:36] "Initialized new in-memory state store"
May 20 13:50:22.356807 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.356795    2550 policy_none.go:49] "None policy: Start"
May 20 13:50:22.356863 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.356817    2550 memory_manager.go:186] "Starting memorymanager" policy="None"
May 20 13:50:22.356863 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.356827    2550 state_mem.go:35] "Initializing new in-memory state store"
May 20 13:50:22.400213 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.390069    2550 manager.go:341] "Starting Device Plugin manager"
May 20 13:50:22.400213 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.390106    2550 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 20 13:50:22.400213 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.390119    2550 server.go:85] "Starting device plugin registration server"
May 20 13:50:22.400213 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.390352    2550 eviction_manager.go:189] "Eviction manager: starting control loop"
May 20 13:50:22.400213 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.390364    2550 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 20 13:50:22.400213 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.390449    2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 20 13:50:22.400213 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.390529    2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 20 13:50:22.400213 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.390539    2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 20 13:50:22.400213 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.391043    2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 20 13:50:22.400213 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.391090    2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:22.460663 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.460623    2550 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 20 13:50:22.461817 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.461794    2550 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 20 13:50:22.461817 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.461816    2550 status_manager.go:230] "Starting to sync pod status with apiserver"
May 20 13:50:22.461959 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.461845    2550 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 20 13:50:22.461959 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.461853    2550 kubelet.go:2451] "Starting kubelet main sync loop"
May 20 13:50:22.461959 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.461890    2550 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 20 13:50:22.467183 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.467160    2550 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:22.491387 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.491328    2550 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:22.492695 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.492681    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:22.492768 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.492711    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:22.492768 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.492723    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:22.492839 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.492769    2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-117.ec2.internal"
May 20 13:50:22.501880 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.501866    2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-117.ec2.internal"
May 20 13:50:22.501933 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.501893    2550 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-117.ec2.internal\": node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:22.516088 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.516065    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:22.562410 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.562364    2550 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"]
May 20 13:50:22.562532 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.562477    2550 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:22.564528 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.564511    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:22.564631 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.564540    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:22.564631 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.564551    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:22.565782 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.565769    2550 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:22.565950 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.565934    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.566016 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.565970    2550 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:22.566527 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.566513    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:22.566597 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.566537    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:22.566597 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.566552    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:22.566672 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.566596    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:22.566672 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.566617    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:22.566672 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.566627    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:22.567730 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.567715    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.567790 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.567745    2550 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 13:50:22.568482 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.568467    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientMemory"
May 20 13:50:22.568546 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.568497    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasNoDiskPressure"
May 20 13:50:22.568546 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.568508    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeHasSufficientPID"
May 20 13:50:22.583924 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.583902    2550 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-117.ec2.internal\" not found" node="ip-10-0-129-117.ec2.internal"
May 20 13:50:22.587978 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.587960    2550 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-117.ec2.internal\" not found" node="ip-10-0-129-117.ec2.internal"
May 20 13:50:22.616787 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.616759    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:22.636727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.636692    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.636820 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.636732    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.636820 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.636756    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f81a807c436a78d0ae12cc5e66b2e05-config\") pod \"kube-apiserver-proxy-ip-10-0-129-117.ec2.internal\" (UID: \"6f81a807c436a78d0ae12cc5e66b2e05\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.717133 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.717106    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:22.737457 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.737432    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.737518 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.737466    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.737518 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.737491    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f81a807c436a78d0ae12cc5e66b2e05-config\") pod \"kube-apiserver-proxy-ip-10-0-129-117.ec2.internal\" (UID: \"6f81a807c436a78d0ae12cc5e66b2e05\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.737579 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.737537    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.737579 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.737547    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6f81a807c436a78d0ae12cc5e66b2e05-config\") pod \"kube-apiserver-proxy-ip-10-0-129-117.ec2.internal\" (UID: \"6f81a807c436a78d0ae12cc5e66b2e05\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.737579 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.737554    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a720ccbb8d80c2c23e6edde668812484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal\" (UID: \"a720ccbb8d80c2c23e6edde668812484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.817892 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.817813    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:22.886298 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.886260    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.890018 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:22.890000    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 13:50:22.918370 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:22.918327    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:23.018908 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:23.018862    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:23.119433 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:23.119362    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:23.220010 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:23.219962    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:23.241381 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.241354    2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 20 13:50:23.241990 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.241495    2550 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 13:50:23.241990 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.241534    2550 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 13:50:23.266582 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.266557    2550 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:23.320105 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:23.320070    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:23.321173 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.321129    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 13:45:22 +0000 UTC" deadline="2027-11-03 05:40:32.753386906 +0000 UTC"
May 20 13:50:23.321243 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.321175    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12759h50m9.432216869s"
May 20 13:50:23.335420 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.335390    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 20 13:50:23.347315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.347289    2550 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 13:50:23.368039 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.368012    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hf6xr"
May 20 13:50:23.377282 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.377223    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hf6xr"
May 20 13:50:23.390446 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:23.390415    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda720ccbb8d80c2c23e6edde668812484.slice/crio-ee7e3f6c2827ca3fa6796461b3cfc9285576902457e464eed10c41da11266b81 WatchSource:0}: Error finding container ee7e3f6c2827ca3fa6796461b3cfc9285576902457e464eed10c41da11266b81: Status 404 returned error can't find the container with id ee7e3f6c2827ca3fa6796461b3cfc9285576902457e464eed10c41da11266b81
May 20 13:50:23.390671 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:23.390652    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f81a807c436a78d0ae12cc5e66b2e05.slice/crio-4da4f3c142621ccc0a7633eda5faf431e470e4fdffc4a35bf95100f432f38afc WatchSource:0}: Error finding container 4da4f3c142621ccc0a7633eda5faf431e470e4fdffc4a35bf95100f432f38afc: Status 404 returned error can't find the container with id 4da4f3c142621ccc0a7633eda5faf431e470e4fdffc4a35bf95100f432f38afc
May 20 13:50:23.396621 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.396606    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 13:50:23.420916 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:23.420884    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:23.465252 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.465202    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal" event={"ID":"6f81a807c436a78d0ae12cc5e66b2e05","Type":"ContainerStarted","Data":"4da4f3c142621ccc0a7633eda5faf431e470e4fdffc4a35bf95100f432f38afc"}
May 20 13:50:23.466031 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.466006    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" event={"ID":"a720ccbb8d80c2c23e6edde668812484","Type":"ContainerStarted","Data":"ee7e3f6c2827ca3fa6796461b3cfc9285576902457e464eed10c41da11266b81"}
May 20 13:50:23.521230 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:23.521178    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:23.621716 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:23.621684    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:23.722250 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:23.722170    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:23.822981 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:23.822942    2550 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-129-117.ec2.internal\" not found"
May 20 13:50:23.900077 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.900042    2550 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:23.936365 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.936319    2550 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal"
May 20 13:50:23.956513 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.956485    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 13:50:23.957387 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.957364    2550 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal"
May 20 13:50:23.971959 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:23.971933    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 13:50:24.278998 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.278969    2550 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:24.309437 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.309407    2550 apiserver.go:52] "Watching apiserver"
May 20 13:50:24.319000 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.318974    2550 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 20 13:50:24.319399 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.319371    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal","openshift-multus/multus-49mt5","openshift-network-diagnostics/network-check-target-7hds7","openshift-network-operator/iptables-alerter-th9rk","kube-system/konnectivity-agent-t2fbl","openshift-dns/node-resolver-rcsnb","openshift-multus/multus-additional-cni-plugins-qx4t2","openshift-multus/network-metrics-daemon-rvjzj","openshift-ovn-kubernetes/ovnkube-node-jbcdp","kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6","openshift-cluster-node-tuning-operator/tuned-mqkqs","openshift-image-registry/node-ca-2gc7l"]
May 20 13:50:24.320855 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.320822    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.322860 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.322834    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49mt5"
May 20 13:50:24.323455 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.323434    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 20 13:50:24.323543 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.323470    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 20 13:50:24.323543 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.323486    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 20 13:50:24.323659 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.323643    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 20 13:50:24.323763 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.323747    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kcbqn\""
May 20 13:50:24.323922 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.323906    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:24.323979 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.323963    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 20 13:50:24.324031 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:24.323971    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:24.324088 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.324044    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-th9rk"
May 20 13:50:24.325084 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.325062    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:24.325598 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.325579    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 20 13:50:24.325709 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.325690    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lckns\""
May 20 13:50:24.326446 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.326176    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rcsnb"
May 20 13:50:24.326734 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.326713    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 20 13:50:24.326827 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.326760    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 20 13:50:24.327106 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.326950    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 20 13:50:24.327106 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.326958    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2xvnw\""
May 20 13:50:24.327474 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.327450    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:24.327562 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:24.327536    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:24.327892 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.327864    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 20 13:50:24.328012 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.327920    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 20 13:50:24.328110 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.328092    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qh7ft\""
May 20 13:50:24.328658 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.328623    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 20 13:50:24.328856 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.328839    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.328969 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.328944    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rltdz\""
May 20 13:50:24.329578 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.329559    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 20 13:50:24.331628 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.331603    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 20 13:50:24.332123 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.332105    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.332354 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.332318    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 20 13:50:24.332562 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.332539    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 20 13:50:24.332661 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.332615    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hvvlr\""
May 20 13:50:24.333415 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.333296    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 20 13:50:24.333792 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.333507    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 20 13:50:24.333792 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.333574    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 20 13:50:24.333925 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.333837    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.335147 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.335126    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 20 13:50:24.335231 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.335171    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 20 13:50:24.335524 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.335497    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 20 13:50:24.336411 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.335839    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rllbv\""
May 20 13:50:24.336515 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.336496    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m5jvj\""
May 20 13:50:24.336606 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.336590    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 20 13:50:24.336787 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.336771    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2gc7l"
May 20 13:50:24.337429 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.337168    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 20 13:50:24.338894 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.338875    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 20 13:50:24.339061 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.339043    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 20 13:50:24.340378 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.340360    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8snb7\""
May 20 13:50:24.340668 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.340651    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 20 13:50:24.344969 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.344949    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f08a998-c27c-422e-bfe3-697e2be370d2-konnectivity-ca\") pod \"konnectivity-agent-t2fbl\" (UID: \"5f08a998-c27c-422e-bfe3-697e2be370d2\") " pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:24.345073 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.344982    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-cni-bin\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.345073 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345001    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-os-release\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.345073 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345023    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-etc-kubernetes\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.345226 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345085    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncl42\" (UniqueName: \"kubernetes.io/projected/05155843-11d8-40f1-ad79-012c0cd08e83-kube-api-access-ncl42\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.345226 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345136    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz24f\" (UniqueName: \"kubernetes.io/projected/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-kube-api-access-bz24f\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:24.345226 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345164    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-cnibin\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.345226 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345205    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-cnibin\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.345439 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345239    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-run-multus-certs\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.345439 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345284    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-sysctl-d\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.345439 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345314    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-run-netns\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.345439 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345348    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-run-openvswitch\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.345439 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345374    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-log-socket\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.345439 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345413    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.345727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345453    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-registration-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.345727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345485    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27731076-9ec4-4347-8b2f-0baf5b1e8622-host\") pod \"node-ca-2gc7l\" (UID: \"27731076-9ec4-4347-8b2f-0baf5b1e8622\") " pod="openshift-image-registry/node-ca-2gc7l"
May 20 13:50:24.345727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345513    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/576d8a0a-be91-4189-b61a-3cec8a8834f2-hosts-file\") pod \"node-resolver-rcsnb\" (UID: \"576d8a0a-be91-4189-b61a-3cec8a8834f2\") " pod="openshift-dns/node-resolver-rcsnb"
May 20 13:50:24.345727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345546    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-sysconfig\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.345727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345573    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f08a998-c27c-422e-bfe3-697e2be370d2-agent-certs\") pod \"konnectivity-agent-t2fbl\" (UID: \"5f08a998-c27c-422e-bfe3-697e2be370d2\") " pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:24.345727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345587    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-etc-openvswitch\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.345727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345605    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-system-cni-dir\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.345727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345627    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6502c66b-dfda-470b-98f8-6ec092bcfec2-cni-binary-copy\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.345727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345660    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6502c66b-dfda-470b-98f8-6ec092bcfec2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.345727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345696    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-etc-selinux\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.345727 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345720    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345740    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-kubelet\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345762    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345799    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-sysctl-conf\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345821    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-lib-modules\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345844    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bktrj\" (UniqueName: \"kubernetes.io/projected/85250934-bd07-4e14-92bd-31c11cccc2b1-kube-api-access-bktrj\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345867    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddq4f\" (UniqueName: \"kubernetes.io/projected/27731076-9ec4-4347-8b2f-0baf5b1e8622-kube-api-access-ddq4f\") pod \"node-ca-2gc7l\" (UID: \"27731076-9ec4-4347-8b2f-0baf5b1e8622\") " pod="openshift-image-registry/node-ca-2gc7l"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345890    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-slash\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345911    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-run-k8s-cni-cncf-io\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.345956    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/148ba522-00f2-49f0-b1b2-0d3c8860b2eb-iptables-alerter-script\") pod \"iptables-alerter-th9rk\" (UID: \"148ba522-00f2-49f0-b1b2-0d3c8860b2eb\") " pod="openshift-network-operator/iptables-alerter-th9rk"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346005    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4d7\" (UniqueName: \"kubernetes.io/projected/148ba522-00f2-49f0-b1b2-0d3c8860b2eb-kube-api-access-dp4d7\") pod \"iptables-alerter-th9rk\" (UID: \"148ba522-00f2-49f0-b1b2-0d3c8860b2eb\") " pod="openshift-network-operator/iptables-alerter-th9rk"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346058    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-node-log\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346093    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6502c66b-dfda-470b-98f8-6ec092bcfec2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346118    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-var-lib-cni-multus\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346195    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.346237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346225    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-sys-fs\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346262    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-kubernetes\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346295    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346351    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-ovnkube-config\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346384    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-run-netns\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346409    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st6jg\" (UniqueName: \"kubernetes.io/projected/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-kube-api-access-st6jg\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346432    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-var-lib-kubelet\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346455    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85250934-bd07-4e14-92bd-31c11cccc2b1-tmp\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346483    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbnv5\" (UniqueName: \"kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5\") pod \"network-check-target-7hds7\" (UID: \"8d2b9089-b3ec-4941-91cc-7c3cada6db01\") " pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346507    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/148ba522-00f2-49f0-b1b2-0d3c8860b2eb-host-slash\") pod \"iptables-alerter-th9rk\" (UID: \"148ba522-00f2-49f0-b1b2-0d3c8860b2eb\") " pod="openshift-network-operator/iptables-alerter-th9rk"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346531    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfnfg\" (UniqueName: \"kubernetes.io/projected/576d8a0a-be91-4189-b61a-3cec8a8834f2-kube-api-access-mfnfg\") pod \"node-resolver-rcsnb\" (UID: \"576d8a0a-be91-4189-b61a-3cec8a8834f2\") " pod="openshift-dns/node-resolver-rcsnb"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346556    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfcr\" (UniqueName: \"kubernetes.io/projected/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-kube-api-access-kkfcr\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346578    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-systemd\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346612    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-run-systemd\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346636    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-tuned\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346657    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27731076-9ec4-4347-8b2f-0baf5b1e8622-serviceca\") pod \"node-ca-2gc7l\" (UID: \"27731076-9ec4-4347-8b2f-0baf5b1e8622\") " pod="openshift-image-registry/node-ca-2gc7l"
May 20 13:50:24.346927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346679    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-run-ovn\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346700    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-system-cni-dir\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346724    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-os-release\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346772    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-var-lib-kubelet\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346803    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-hostroot\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346830    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-multus-conf-dir\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346859    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576d8a0a-be91-4189-b61a-3cec8a8834f2-tmp-dir\") pod \"node-resolver-rcsnb\" (UID: \"576d8a0a-be91-4189-b61a-3cec8a8834f2\") " pod="openshift-dns/node-resolver-rcsnb"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346900    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-systemd-units\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346946    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-multus-cni-dir\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.346972    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05155843-11d8-40f1-ad79-012c0cd08e83-cni-binary-copy\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347008    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-var-lib-cni-bin\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347052    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-run\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347095    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-sys\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347145    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-var-lib-openvswitch\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347181    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-cni-netd\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347241    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-env-overrides\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347303    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-ovnkube-script-lib\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.347770 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347329    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f7sr\" (UniqueName: \"kubernetes.io/projected/6502c66b-dfda-470b-98f8-6ec092bcfec2-kube-api-access-8f7sr\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.348616 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347390    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-multus-socket-dir-parent\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.348616 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347420    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-modprobe-d\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.348616 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347444    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-host\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.348616 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347467    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-ovn-node-metrics-cert\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.348616 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347486    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05155843-11d8-40f1-ad79-012c0cd08e83-multus-daemon-config\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.348616 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347521    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-socket-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.348616 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.347536    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-device-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.378114 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.378078    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:23 +0000 UTC" deadline="2028-01-06 07:21:03.852840988 +0000 UTC"
May 20 13:50:24.378114 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.378111    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14297h30m39.474732813s"
May 20 13:50:24.436361 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.436309    2550 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 20 13:50:24.449099 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449065    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bz24f\" (UniqueName: \"kubernetes.io/projected/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-kube-api-access-bz24f\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:24.449099 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449108    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-cnibin\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.449331 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449166    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-cnibin\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.449331 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449239    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-cnibin\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.449331 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449317    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-run-multus-certs\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.449501 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449389    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-sysctl-d\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.449501 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449414    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-run-multus-certs\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.449501 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449414    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-cnibin\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.449501 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449419    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-run-netns\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.449501 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449463    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-run-netns\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.449722 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449504    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-run-openvswitch\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.449722 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449541    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-sysctl-d\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.449722 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449548    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-log-socket\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.449722 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449591    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-run-openvswitch\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.449722 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449577    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.449722 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449640    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-registration-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.449722 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449640    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.449722 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449683    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-log-socket\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.449722 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449685    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-registration-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449716    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27731076-9ec4-4347-8b2f-0baf5b1e8622-host\") pod \"node-ca-2gc7l\" (UID: \"27731076-9ec4-4347-8b2f-0baf5b1e8622\") " pod="openshift-image-registry/node-ca-2gc7l"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449794    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27731076-9ec4-4347-8b2f-0baf5b1e8622-host\") pod \"node-ca-2gc7l\" (UID: \"27731076-9ec4-4347-8b2f-0baf5b1e8622\") " pod="openshift-image-registry/node-ca-2gc7l"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449813    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/576d8a0a-be91-4189-b61a-3cec8a8834f2-hosts-file\") pod \"node-resolver-rcsnb\" (UID: \"576d8a0a-be91-4189-b61a-3cec8a8834f2\") " pod="openshift-dns/node-resolver-rcsnb"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449833    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-sysconfig\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449852    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f08a998-c27c-422e-bfe3-697e2be370d2-agent-certs\") pod \"konnectivity-agent-t2fbl\" (UID: \"5f08a998-c27c-422e-bfe3-697e2be370d2\") " pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449875    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-etc-openvswitch\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449914    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-sysconfig\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449915    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-etc-openvswitch\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449931    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-system-cni-dir\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449960    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6502c66b-dfda-470b-98f8-6ec092bcfec2-cni-binary-copy\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449988    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/576d8a0a-be91-4189-b61a-3cec8a8834f2-hosts-file\") pod \"node-resolver-rcsnb\" (UID: \"576d8a0a-be91-4189-b61a-3cec8a8834f2\") " pod="openshift-dns/node-resolver-rcsnb"
May 20 13:50:24.450001 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449994    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-system-cni-dir\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.449991    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6502c66b-dfda-470b-98f8-6ec092bcfec2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450053    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-etc-selinux\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450081    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450155    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-kubelet\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450178    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450209    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-sysctl-conf\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450240    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-lib-modules\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450266    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-kubelet\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450180    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-etc-selinux\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450286    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bktrj\" (UniqueName: \"kubernetes.io/projected/85250934-bd07-4e14-92bd-31c11cccc2b1-kube-api-access-bktrj\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:24.450305    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450327    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ddq4f\" (UniqueName: \"kubernetes.io/projected/27731076-9ec4-4347-8b2f-0baf5b1e8622-kube-api-access-ddq4f\") pod \"node-ca-2gc7l\" (UID: \"27731076-9ec4-4347-8b2f-0baf5b1e8622\") " pod="openshift-image-registry/node-ca-2gc7l"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450411    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-slash\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450463    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-slash\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.450529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450472    2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450609    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450675    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6502c66b-dfda-470b-98f8-6ec092bcfec2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450688    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-sysctl-conf\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450662    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6502c66b-dfda-470b-98f8-6ec092bcfec2-cni-binary-copy\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:24.450806    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs podName:a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be nodeName:}" failed. No retries permitted until 2026-05-20 13:50:24.950426391 +0000 UTC m=+3.076809539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs") pod "network-metrics-daemon-rvjzj" (UID: "a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450848    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-lib-modules\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450861    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-run-k8s-cni-cncf-io\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450887    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-run-k8s-cni-cncf-io\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450923    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/148ba522-00f2-49f0-b1b2-0d3c8860b2eb-iptables-alerter-script\") pod \"iptables-alerter-th9rk\" (UID: \"148ba522-00f2-49f0-b1b2-0d3c8860b2eb\") " pod="openshift-network-operator/iptables-alerter-th9rk"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450943    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4d7\" (UniqueName: \"kubernetes.io/projected/148ba522-00f2-49f0-b1b2-0d3c8860b2eb-kube-api-access-dp4d7\") pod \"iptables-alerter-th9rk\" (UID: \"148ba522-00f2-49f0-b1b2-0d3c8860b2eb\") " pod="openshift-network-operator/iptables-alerter-th9rk"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.450966    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-node-log\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451049    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6502c66b-dfda-470b-98f8-6ec092bcfec2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451145    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-var-lib-cni-multus\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.451155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451159    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-node-log\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451181    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451217    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-sys-fs\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451251    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-kubernetes\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451281    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451311    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-ovnkube-config\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451315    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-kubelet-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451369    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-run-netns\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451397    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-st6jg\" (UniqueName: \"kubernetes.io/projected/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-kube-api-access-st6jg\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451433    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-kubernetes\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451431    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-var-lib-kubelet\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451479    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85250934-bd07-4e14-92bd-31c11cccc2b1-tmp\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451497    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-sys-fs\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451522    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnv5\" (UniqueName: \"kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5\") pod \"network-check-target-7hds7\" (UID: \"8d2b9089-b3ec-4941-91cc-7c3cada6db01\") " pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451548    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/148ba522-00f2-49f0-b1b2-0d3c8860b2eb-host-slash\") pod \"iptables-alerter-th9rk\" (UID: \"148ba522-00f2-49f0-b1b2-0d3c8860b2eb\") " pod="openshift-network-operator/iptables-alerter-th9rk"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451590    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/148ba522-00f2-49f0-b1b2-0d3c8860b2eb-iptables-alerter-script\") pod \"iptables-alerter-th9rk\" (UID: \"148ba522-00f2-49f0-b1b2-0d3c8860b2eb\") " pod="openshift-network-operator/iptables-alerter-th9rk"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451604    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mfnfg\" (UniqueName: \"kubernetes.io/projected/576d8a0a-be91-4189-b61a-3cec8a8834f2-kube-api-access-mfnfg\") pod \"node-resolver-rcsnb\" (UID: \"576d8a0a-be91-4189-b61a-3cec8a8834f2\") " pod="openshift-dns/node-resolver-rcsnb"
May 20 13:50:24.451791 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451642    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfcr\" (UniqueName: \"kubernetes.io/projected/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-kube-api-access-kkfcr\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451665    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-systemd\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451707    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-run-systemd\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451728    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-tuned\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451751    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27731076-9ec4-4347-8b2f-0baf5b1e8622-serviceca\") pod \"node-ca-2gc7l\" (UID: \"27731076-9ec4-4347-8b2f-0baf5b1e8622\") " pod="openshift-image-registry/node-ca-2gc7l"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451775    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-run-ovn\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451796    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451801    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-system-cni-dir\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451845    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-ovnkube-config\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451859    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-var-lib-cni-multus\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451852    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-os-release\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451892    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-run-netns\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451960    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-os-release\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.451955    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-var-lib-kubelet\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452031    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-var-lib-kubelet\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452178    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-run-ovn\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452245    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/148ba522-00f2-49f0-b1b2-0d3c8860b2eb-host-slash\") pod \"iptables-alerter-th9rk\" (UID: \"148ba522-00f2-49f0-b1b2-0d3c8860b2eb\") " pod="openshift-network-operator/iptables-alerter-th9rk"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452309    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-systemd\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.452450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452327    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-var-lib-kubelet\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452392    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-system-cni-dir\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452396    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-run-systemd\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452409    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/27731076-9ec4-4347-8b2f-0baf5b1e8622-serviceca\") pod \"node-ca-2gc7l\" (UID: \"27731076-9ec4-4347-8b2f-0baf5b1e8622\") " pod="openshift-image-registry/node-ca-2gc7l"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452477    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-hostroot\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452504    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-multus-conf-dir\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452531    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576d8a0a-be91-4189-b61a-3cec8a8834f2-tmp-dir\") pod \"node-resolver-rcsnb\" (UID: \"576d8a0a-be91-4189-b61a-3cec8a8834f2\") " pod="openshift-dns/node-resolver-rcsnb"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452534    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-hostroot\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452586    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-systemd-units\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452585    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-multus-conf-dir\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452607    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-multus-cni-dir\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452690    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-multus-cni-dir\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452719    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-systemd-units\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452792    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05155843-11d8-40f1-ad79-012c0cd08e83-cni-binary-copy\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453120    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-var-lib-cni-bin\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453191    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-run\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453224    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-sys\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453271    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-var-lib-openvswitch\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.453315 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453316    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-cni-netd\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453369    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-env-overrides\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453404    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-ovnkube-script-lib\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453445    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7sr\" (UniqueName: \"kubernetes.io/projected/6502c66b-dfda-470b-98f8-6ec092bcfec2-kube-api-access-8f7sr\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453479    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-multus-socket-dir-parent\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453517    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-modprobe-d\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453545    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-host\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453579    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-ovn-node-metrics-cert\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453613    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05155843-11d8-40f1-ad79-012c0cd08e83-multus-daemon-config\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453649    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-socket-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453682    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-device-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453716    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f08a998-c27c-422e-bfe3-697e2be370d2-konnectivity-ca\") pod \"konnectivity-agent-t2fbl\" (UID: \"5f08a998-c27c-422e-bfe3-697e2be370d2\") " pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.453747    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-cni-bin\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.454085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.452820    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6502c66b-dfda-470b-98f8-6ec092bcfec2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.454655 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.454416    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f08a998-c27c-422e-bfe3-697e2be370d2-agent-certs\") pod \"konnectivity-agent-t2fbl\" (UID: \"5f08a998-c27c-422e-bfe3-697e2be370d2\") " pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:24.454655 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.454482    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-os-release\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.454655 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.454578    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6502c66b-dfda-470b-98f8-6ec092bcfec2-os-release\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.454655 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.454629    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-cni-bin\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.454818 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.454689    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-host-var-lib-cni-bin\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.454818 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.454748    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-run\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.454818 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.454797    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-sys\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.454948 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.454841    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-var-lib-openvswitch\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.454948 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.454890    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-host-cni-netd\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.454948 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.454933    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f08a998-c27c-422e-bfe3-697e2be370d2-konnectivity-ca\") pod \"konnectivity-agent-t2fbl\" (UID: \"5f08a998-c27c-422e-bfe3-697e2be370d2\") " pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:24.455119 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.455086    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-host\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.455278 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.455258    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-env-overrides\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.456444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.455586    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-multus-socket-dir-parent\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.456444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.455602    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-ovnkube-script-lib\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.456444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.455731    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-socket-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.456444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.455792    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576d8a0a-be91-4189-b61a-3cec8a8834f2-tmp-dir\") pod \"node-resolver-rcsnb\" (UID: \"576d8a0a-be91-4189-b61a-3cec8a8834f2\") " pod="openshift-dns/node-resolver-rcsnb"
May 20 13:50:24.456444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.455830    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-modprobe-d\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.456444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.455875    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-device-dir\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.456444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.455908    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-etc-kubernetes\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.456444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.456016    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05155843-11d8-40f1-ad79-012c0cd08e83-etc-kubernetes\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.456444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.456060    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ncl42\" (UniqueName: \"kubernetes.io/projected/05155843-11d8-40f1-ad79-012c0cd08e83-kube-api-access-ncl42\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.456444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.456178    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05155843-11d8-40f1-ad79-012c0cd08e83-multus-daemon-config\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.456444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.456385    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05155843-11d8-40f1-ad79-012c0cd08e83-cni-binary-copy\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.457026 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.456620    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/85250934-bd07-4e14-92bd-31c11cccc2b1-etc-tuned\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.457919 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.457832    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-ovn-node-metrics-cert\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.458566 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.458547    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85250934-bd07-4e14-92bd-31c11cccc2b1-tmp\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.460602 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.460583    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz24f\" (UniqueName: \"kubernetes.io/projected/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-kube-api-access-bz24f\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:24.462621 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:24.462599    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:24.462727 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:24.462627    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:24.462727 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:24.462641    2550 projected.go:194] Error preparing data for projected volume kube-api-access-xbnv5 for pod openshift-network-diagnostics/network-check-target-7hds7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:24.462727 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:24.462709    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5 podName:8d2b9089-b3ec-4941-91cc-7c3cada6db01 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:24.962688584 +0000 UTC m=+3.089071741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xbnv5" (UniqueName: "kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5") pod "network-check-target-7hds7" (UID: "8d2b9089-b3ec-4941-91cc-7c3cada6db01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:24.464425 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.464364    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfnfg\" (UniqueName: \"kubernetes.io/projected/576d8a0a-be91-4189-b61a-3cec8a8834f2-kube-api-access-mfnfg\") pod \"node-resolver-rcsnb\" (UID: \"576d8a0a-be91-4189-b61a-3cec8a8834f2\") " pod="openshift-dns/node-resolver-rcsnb"
May 20 13:50:24.464425 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.464364    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddq4f\" (UniqueName: \"kubernetes.io/projected/27731076-9ec4-4347-8b2f-0baf5b1e8622-kube-api-access-ddq4f\") pod \"node-ca-2gc7l\" (UID: \"27731076-9ec4-4347-8b2f-0baf5b1e8622\") " pod="openshift-image-registry/node-ca-2gc7l"
May 20 13:50:24.466952 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.466840    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f7sr\" (UniqueName: \"kubernetes.io/projected/6502c66b-dfda-470b-98f8-6ec092bcfec2-kube-api-access-8f7sr\") pod \"multus-additional-cni-plugins-qx4t2\" (UID: \"6502c66b-dfda-470b-98f8-6ec092bcfec2\") " pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.466952 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.466913    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfcr\" (UniqueName: \"kubernetes.io/projected/b7b7bb27-c274-4fa1-a5fb-97fe855429c3-kube-api-access-kkfcr\") pod \"ovnkube-node-jbcdp\" (UID: \"b7b7bb27-c274-4fa1-a5fb-97fe855429c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.467175 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.467138    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4d7\" (UniqueName: \"kubernetes.io/projected/148ba522-00f2-49f0-b1b2-0d3c8860b2eb-kube-api-access-dp4d7\") pod \"iptables-alerter-th9rk\" (UID: \"148ba522-00f2-49f0-b1b2-0d3c8860b2eb\") " pod="openshift-network-operator/iptables-alerter-th9rk"
May 20 13:50:24.467771 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.467746    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bktrj\" (UniqueName: \"kubernetes.io/projected/85250934-bd07-4e14-92bd-31c11cccc2b1-kube-api-access-bktrj\") pod \"tuned-mqkqs\" (UID: \"85250934-bd07-4e14-92bd-31c11cccc2b1\") " pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.468763 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.468742    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncl42\" (UniqueName: \"kubernetes.io/projected/05155843-11d8-40f1-ad79-012c0cd08e83-kube-api-access-ncl42\") pod \"multus-49mt5\" (UID: \"05155843-11d8-40f1-ad79-012c0cd08e83\") " pod="openshift-multus/multus-49mt5"
May 20 13:50:24.468848 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.468779    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6jg\" (UniqueName: \"kubernetes.io/projected/f7dbfe28-779d-4c80-aa72-8e0cd8af4385-kube-api-access-st6jg\") pod \"aws-ebs-csi-driver-node-shnw6\" (UID: \"f7dbfe28-779d-4c80-aa72-8e0cd8af4385\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.499775 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.499742    2550 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 13:50:24.636046 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.635961    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qx4t2"
May 20 13:50:24.643263 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.643234    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49mt5"
May 20 13:50:24.651974 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.651944    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-th9rk"
May 20 13:50:24.657708 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.657675    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:24.664444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.664418    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rcsnb"
May 20 13:50:24.670190 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.670169    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:24.676869 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.676848    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6"
May 20 13:50:24.683520 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.683503    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mqkqs"
May 20 13:50:24.689139 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.689122    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2gc7l"
May 20 13:50:24.929441 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:24.929396    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05155843_11d8_40f1_ad79_012c0cd08e83.slice/crio-7385c4da0dc1521b687bac1a48f6215dd382194681fa00008938fbf11bcaca0a WatchSource:0}: Error finding container 7385c4da0dc1521b687bac1a48f6215dd382194681fa00008938fbf11bcaca0a: Status 404 returned error can't find the container with id 7385c4da0dc1521b687bac1a48f6215dd382194681fa00008938fbf11bcaca0a
May 20 13:50:24.930998 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:24.930962    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7dbfe28_779d_4c80_aa72_8e0cd8af4385.slice/crio-7578745ea7428c0c95aa6f75fea8e395753f66a7e23a0856bfc89d9c5e055d01 WatchSource:0}: Error finding container 7578745ea7428c0c95aa6f75fea8e395753f66a7e23a0856bfc89d9c5e055d01: Status 404 returned error can't find the container with id 7578745ea7428c0c95aa6f75fea8e395753f66a7e23a0856bfc89d9c5e055d01
May 20 13:50:24.932164 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:24.932139    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f08a998_c27c_422e_bfe3_697e2be370d2.slice/crio-d5fc511ff902d4b30ce7d65112a7df582c44abaaddd449d97c2f9332af495782 WatchSource:0}: Error finding container d5fc511ff902d4b30ce7d65112a7df582c44abaaddd449d97c2f9332af495782: Status 404 returned error can't find the container with id d5fc511ff902d4b30ce7d65112a7df582c44abaaddd449d97c2f9332af495782
May 20 13:50:24.935865 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:24.935838    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6502c66b_dfda_470b_98f8_6ec092bcfec2.slice/crio-8a83ab4ff689fc980ed1fe25dbfeb7b53d41aa22656bd9ffc7a6d4b866705477 WatchSource:0}: Error finding container 8a83ab4ff689fc980ed1fe25dbfeb7b53d41aa22656bd9ffc7a6d4b866705477: Status 404 returned error can't find the container with id 8a83ab4ff689fc980ed1fe25dbfeb7b53d41aa22656bd9ffc7a6d4b866705477
May 20 13:50:24.936747 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:50:24.936728    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148ba522_00f2_49f0_b1b2_0d3c8860b2eb.slice/crio-5540fb51de685e7d372076048a06eaba83d383edc5e9e228de31e172ad3f81b4 WatchSource:0}: Error finding container 5540fb51de685e7d372076048a06eaba83d383edc5e9e228de31e172ad3f81b4: Status 404 returned error can't find the container with id 5540fb51de685e7d372076048a06eaba83d383edc5e9e228de31e172ad3f81b4
May 20 13:50:24.960470 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:24.960281    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:24.960600 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:24.960435    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:24.960600 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:24.960594    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs podName:a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be nodeName:}" failed. No retries permitted until 2026-05-20 13:50:25.960575518 +0000 UTC m=+4.086958655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs") pod "network-metrics-daemon-rvjzj" (UID: "a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:25.061705 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.061673    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnv5\" (UniqueName: \"kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5\") pod \"network-check-target-7hds7\" (UID: \"8d2b9089-b3ec-4941-91cc-7c3cada6db01\") " pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:25.061857 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:25.061834    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:25.061902 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:25.061858    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:25.061902 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:25.061871    2550 projected.go:194] Error preparing data for projected volume kube-api-access-xbnv5 for pod openshift-network-diagnostics/network-check-target-7hds7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:25.061966 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:25.061936    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5 podName:8d2b9089-b3ec-4941-91cc-7c3cada6db01 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:26.061917345 +0000 UTC m=+4.188300496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbnv5" (UniqueName: "kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5") pod "network-check-target-7hds7" (UID: "8d2b9089-b3ec-4941-91cc-7c3cada6db01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:25.379126 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.378965    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 13:45:23 +0000 UTC" deadline="2027-12-19 22:17:57.74938344 +0000 UTC"
May 20 13:50:25.379126 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.379054    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13880h27m32.370334892s"
May 20 13:50:25.463155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.462636    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:25.463155 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:25.462767    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:25.481037 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.480979    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49mt5" event={"ID":"05155843-11d8-40f1-ad79-012c0cd08e83","Type":"ContainerStarted","Data":"7385c4da0dc1521b687bac1a48f6215dd382194681fa00008938fbf11bcaca0a"}
May 20 13:50:25.484879 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.484817    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp" event={"ID":"b7b7bb27-c274-4fa1-a5fb-97fe855429c3","Type":"ContainerStarted","Data":"1b5589e7cc72f125701bd73dd61a533fb8485a0a091db0dd007ee72da31c25b1"}
May 20 13:50:25.492396 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.492326    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mqkqs" event={"ID":"85250934-bd07-4e14-92bd-31c11cccc2b1","Type":"ContainerStarted","Data":"1b25d3370d0e5821faebd3044a6f84e17b96f092d7048aad309f348f8863326a"}
May 20 13:50:25.494113 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.494049    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2gc7l" event={"ID":"27731076-9ec4-4347-8b2f-0baf5b1e8622","Type":"ContainerStarted","Data":"72aff5930daa211c7cd585b5de3d94fd0101e08036d023325755493ac6690630"}
May 20 13:50:25.497103 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.497041    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t2fbl" event={"ID":"5f08a998-c27c-422e-bfe3-697e2be370d2","Type":"ContainerStarted","Data":"d5fc511ff902d4b30ce7d65112a7df582c44abaaddd449d97c2f9332af495782"}
May 20 13:50:25.500972 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.500949    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6" event={"ID":"f7dbfe28-779d-4c80-aa72-8e0cd8af4385","Type":"ContainerStarted","Data":"7578745ea7428c0c95aa6f75fea8e395753f66a7e23a0856bfc89d9c5e055d01"}
May 20 13:50:25.509412 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.509071    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal" event={"ID":"6f81a807c436a78d0ae12cc5e66b2e05","Type":"ContainerStarted","Data":"80be902412c468183af229ded128f3e313b4362dad9d93ded7244ee22b5dd904"}
May 20 13:50:25.512548 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.512203    2550 generic.go:358] "Generic (PLEG): container finished" podID="a720ccbb8d80c2c23e6edde668812484" containerID="528a68352aa0b109544d278ead30b06bf56c452cde1fae293e918b2bed27ac44" exitCode=0
May 20 13:50:25.512548 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.512266    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" event={"ID":"a720ccbb8d80c2c23e6edde668812484","Type":"ContainerDied","Data":"528a68352aa0b109544d278ead30b06bf56c452cde1fae293e918b2bed27ac44"}
May 20 13:50:25.522800 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.522744    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rcsnb" event={"ID":"576d8a0a-be91-4189-b61a-3cec8a8834f2","Type":"ContainerStarted","Data":"95f6e08436fbab36ab04003b339d37dfb70b38076f64c7a4b40665710d064b8d"}
May 20 13:50:25.524547 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.524503    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-117.ec2.internal" podStartSLOduration=2.524490139 podStartE2EDuration="2.524490139s" podCreationTimestamp="2026-05-20 13:50:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:25.524115346 +0000 UTC m=+3.650498502" watchObservedRunningTime="2026-05-20 13:50:25.524490139 +0000 UTC m=+3.650873292"
May 20 13:50:25.530624 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.530589    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-th9rk" event={"ID":"148ba522-00f2-49f0-b1b2-0d3c8860b2eb","Type":"ContainerStarted","Data":"5540fb51de685e7d372076048a06eaba83d383edc5e9e228de31e172ad3f81b4"}
May 20 13:50:25.534088 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.534060    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx4t2" event={"ID":"6502c66b-dfda-470b-98f8-6ec092bcfec2","Type":"ContainerStarted","Data":"8a83ab4ff689fc980ed1fe25dbfeb7b53d41aa22656bd9ffc7a6d4b866705477"}
May 20 13:50:25.971021 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:25.970484    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:25.971021 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:25.970626    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:25.971021 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:25.970689    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs podName:a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be nodeName:}" failed. No retries permitted until 2026-05-20 13:50:27.970672071 +0000 UTC m=+6.097055209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs") pod "network-metrics-daemon-rvjzj" (UID: "a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:26.071449 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:26.071360    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnv5\" (UniqueName: \"kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5\") pod \"network-check-target-7hds7\" (UID: \"8d2b9089-b3ec-4941-91cc-7c3cada6db01\") " pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:26.071611 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:26.071551    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:26.071611 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:26.071575    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:26.071611 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:26.071587    2550 projected.go:194] Error preparing data for projected volume kube-api-access-xbnv5 for pod openshift-network-diagnostics/network-check-target-7hds7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:26.071754 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:26.071646    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5 podName:8d2b9089-b3ec-4941-91cc-7c3cada6db01 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:28.071626485 +0000 UTC m=+6.198009616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbnv5" (UniqueName: "kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5") pod "network-check-target-7hds7" (UID: "8d2b9089-b3ec-4941-91cc-7c3cada6db01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:26.465508 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:26.465424    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:26.465972 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:26.465576    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:26.558459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:26.558401    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" event={"ID":"a720ccbb8d80c2c23e6edde668812484","Type":"ContainerStarted","Data":"bdd61020411ebb688863be5f2ab35b5d3f705ee1ef389149f9fe39f913599327"}
May 20 13:50:26.576971 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:26.576173    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-117.ec2.internal" podStartSLOduration=3.576153542 podStartE2EDuration="3.576153542s" podCreationTimestamp="2026-05-20 13:50:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:50:26.576092025 +0000 UTC m=+4.702475178" watchObservedRunningTime="2026-05-20 13:50:26.576153542 +0000 UTC m=+4.702536696"
May 20 13:50:27.464699 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:27.464174    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:27.464699 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:27.464320    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:27.992142 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:27.992102    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:27.992600 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:27.992285    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:27.992657 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:27.992633    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs podName:a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be nodeName:}" failed. No retries permitted until 2026-05-20 13:50:31.992396686 +0000 UTC m=+10.118779821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs") pod "network-metrics-daemon-rvjzj" (UID: "a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:28.093693 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:28.092868    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnv5\" (UniqueName: \"kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5\") pod \"network-check-target-7hds7\" (UID: \"8d2b9089-b3ec-4941-91cc-7c3cada6db01\") " pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:28.093693 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:28.093078    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:28.093693 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:28.093096    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:28.093693 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:28.093110    2550 projected.go:194] Error preparing data for projected volume kube-api-access-xbnv5 for pod openshift-network-diagnostics/network-check-target-7hds7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:28.093693 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:28.093171    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5 podName:8d2b9089-b3ec-4941-91cc-7c3cada6db01 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:32.093152497 +0000 UTC m=+10.219535642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbnv5" (UniqueName: "kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5") pod "network-check-target-7hds7" (UID: "8d2b9089-b3ec-4941-91cc-7c3cada6db01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:28.463079 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:28.462784    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:28.463079 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:28.462976    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:29.462262 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:29.462102    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:29.462772 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:29.462260    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:30.463092 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:30.462613    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:30.463092 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:30.462759    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:31.462969 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:31.462369    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:31.462969 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:31.462508    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:32.029883 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:32.029830    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:32.030088 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:32.030003    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:32.030088 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:32.030079    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs podName:a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be nodeName:}" failed. No retries permitted until 2026-05-20 13:50:40.030058609 +0000 UTC m=+18.156441741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs") pod "network-metrics-daemon-rvjzj" (UID: "a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:32.131085 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:32.131046    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnv5\" (UniqueName: \"kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5\") pod \"network-check-target-7hds7\" (UID: \"8d2b9089-b3ec-4941-91cc-7c3cada6db01\") " pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:32.131260 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:32.131216    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:32.131260 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:32.131242    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:32.131260 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:32.131255    2550 projected.go:194] Error preparing data for projected volume kube-api-access-xbnv5 for pod openshift-network-diagnostics/network-check-target-7hds7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:32.131450 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:32.131323    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5 podName:8d2b9089-b3ec-4941-91cc-7c3cada6db01 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:40.131302686 +0000 UTC m=+18.257686050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbnv5" (UniqueName: "kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5") pod "network-check-target-7hds7" (UID: "8d2b9089-b3ec-4941-91cc-7c3cada6db01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:32.463285 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:32.463203    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:32.463735 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:32.463322    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:33.462772 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:33.462732    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:33.462959 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:33.462885    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:34.462590 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:34.462551    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:34.463054 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:34.462680    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:35.462378 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:35.462331    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:35.462538 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:35.462469    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:36.462379 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:36.462327    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:36.462820 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:36.462477    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:37.462963 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:37.462922    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:37.463401 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:37.463072    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:38.462814 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:38.462720    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:38.462973 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:38.462851    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:39.462636 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:39.462601    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:39.463127 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:39.462740    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:40.083526 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.083492    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:40.083700 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.083661    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:40.083774 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.083736    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs podName:a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be nodeName:}" failed. No retries permitted until 2026-05-20 13:50:56.083715789 +0000 UTC m=+34.210098937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs") pod "network-metrics-daemon-rvjzj" (UID: "a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:40.184023 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.183969    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnv5\" (UniqueName: \"kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5\") pod \"network-check-target-7hds7\" (UID: \"8d2b9089-b3ec-4941-91cc-7c3cada6db01\") " pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:40.184182 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.184112    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:40.184182 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.184130    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:40.184182 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.184142    2550 projected.go:194] Error preparing data for projected volume kube-api-access-xbnv5 for pod openshift-network-diagnostics/network-check-target-7hds7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:40.184322 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.184213    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5 podName:8d2b9089-b3ec-4941-91cc-7c3cada6db01 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:56.18419195 +0000 UTC m=+34.310575098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbnv5" (UniqueName: "kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5") pod "network-check-target-7hds7" (UID: "8d2b9089-b3ec-4941-91cc-7c3cada6db01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:40.218837 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.218806    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-jnqxh"]
May 20 13:50:40.278810 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.278772    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:40.278976 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.278867    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jnqxh" podUID="e7efcced-743d-485e-a499-25e831588966"
May 20 13:50:40.385750 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.385656    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7efcced-743d-485e-a499-25e831588966-dbus\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:40.385750 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.385714    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7efcced-743d-485e-a499-25e831588966-kubelet-config\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:40.385750 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.385742    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:40.463134 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.463097    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:40.463595 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.463216    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:40.486180 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.486146    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7efcced-743d-485e-a499-25e831588966-dbus\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:40.486180 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.486196    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7efcced-743d-485e-a499-25e831588966-kubelet-config\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:40.486494 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.486214    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:40.486494 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.486311    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e7efcced-743d-485e-a499-25e831588966-kubelet-config\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:40.486494 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.486375    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:40.486494 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.486429    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret podName:e7efcced-743d-485e-a499-25e831588966 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:40.98641413 +0000 UTC m=+19.112797261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret") pod "global-pull-secret-syncer-jnqxh" (UID: "e7efcced-743d-485e-a499-25e831588966") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:40.486494 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.486434    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e7efcced-743d-485e-a499-25e831588966-dbus\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:40.990287 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:40.990245    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:40.990492 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.990418    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:40.990551 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:40.990502    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret podName:e7efcced-743d-485e-a499-25e831588966 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:41.990480333 +0000 UTC m=+20.116863468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret") pod "global-pull-secret-syncer-jnqxh" (UID: "e7efcced-743d-485e-a499-25e831588966") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:41.463109 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:41.463068    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:41.463286 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:41.463068    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:41.463286 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:41.463213    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:41.463286 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:41.463270    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jnqxh" podUID="e7efcced-743d-485e-a499-25e831588966"
May 20 13:50:41.999867 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:41.999819    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:42.000045 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:41.999988    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:42.000109 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:42.000065    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret podName:e7efcced-743d-485e-a499-25e831588966 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:44.000047835 +0000 UTC m=+22.126430987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret") pod "global-pull-secret-syncer-jnqxh" (UID: "e7efcced-743d-485e-a499-25e831588966") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:42.463139 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:42.463108    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:42.463274 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:42.463196    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:42.586224 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:42.586194    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t2fbl" event={"ID":"5f08a998-c27c-422e-bfe3-697e2be370d2","Type":"ContainerStarted","Data":"30ab6a9d604ec835c0dfaea725af1fca4e8c156865718452f01e8bcd470e2a00"}
May 20 13:50:43.463073 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.462778    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:43.463222 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.462814    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:43.463222 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:43.463140    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jnqxh" podUID="e7efcced-743d-485e-a499-25e831588966"
May 20 13:50:43.463222 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:43.463183    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:43.589203 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.589169    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2gc7l" event={"ID":"27731076-9ec4-4347-8b2f-0baf5b1e8622","Type":"ContainerStarted","Data":"047ad327f08805f4c259c21304b9517ab4bc5faf72ff8d9bba52930936ad95a5"}
May 20 13:50:43.590670 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.590638    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6" event={"ID":"f7dbfe28-779d-4c80-aa72-8e0cd8af4385","Type":"ContainerStarted","Data":"e4cf3faa2be238c8865834bc8c9768fc73d2cbdae3044043028e25fc1cf9e64f"}
May 20 13:50:43.592020 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.591991    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rcsnb" event={"ID":"576d8a0a-be91-4189-b61a-3cec8a8834f2","Type":"ContainerStarted","Data":"edeb7133d8692bb332a836926d2fa2c3d34d8494f899c9cab03517d4850056cd"}
May 20 13:50:43.593490 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.593453    2550 generic.go:358] "Generic (PLEG): container finished" podID="6502c66b-dfda-470b-98f8-6ec092bcfec2" containerID="b9dad9d4344bedf851d7598880214351fe4020677d89167c2b5d0eeccb220c1d" exitCode=0
May 20 13:50:43.593581 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.593513    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx4t2" event={"ID":"6502c66b-dfda-470b-98f8-6ec092bcfec2","Type":"ContainerDied","Data":"b9dad9d4344bedf851d7598880214351fe4020677d89167c2b5d0eeccb220c1d"}
May 20 13:50:43.595006 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.594970    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49mt5" event={"ID":"05155843-11d8-40f1-ad79-012c0cd08e83","Type":"ContainerStarted","Data":"9a799ba5e7453f2c852ee0ec637c5c70da21f64e6d43b9bcf5d59ba26117d4e7"}
May 20 13:50:43.596448 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.596420    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp" event={"ID":"b7b7bb27-c274-4fa1-a5fb-97fe855429c3","Type":"ContainerStarted","Data":"a6fd80147bc8af6af47d0231b6134aa8d179efbf01506879cfbd85aef0bedb0e"}
May 20 13:50:43.597740 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.597700    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mqkqs" event={"ID":"85250934-bd07-4e14-92bd-31c11cccc2b1","Type":"ContainerStarted","Data":"ae77abfefa5d91965d1dc05941b513800294e4c60fa2f59271e121e7c0edeb1e"}
May 20 13:50:43.607376 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.607320    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2gc7l" podStartSLOduration=4.155446561 podStartE2EDuration="21.607308491s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:50:24.941374776 +0000 UTC m=+3.067757906" lastFinishedPulling="2026-05-20 13:50:42.393236692 +0000 UTC m=+20.519619836" observedRunningTime="2026-05-20 13:50:43.606904776 +0000 UTC m=+21.733287929" watchObservedRunningTime="2026-05-20 13:50:43.607308491 +0000 UTC m=+21.733691644"
May 20 13:50:43.625090 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.625041    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mqkqs" podStartSLOduration=4.105863372 podStartE2EDuration="21.625022541s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:50:24.945006823 +0000 UTC m=+3.071389969" lastFinishedPulling="2026-05-20 13:50:42.464166006 +0000 UTC m=+20.590549138" observedRunningTime="2026-05-20 13:50:43.624730825 +0000 UTC m=+21.751113968" watchObservedRunningTime="2026-05-20 13:50:43.625022541 +0000 UTC m=+21.751405693"
May 20 13:50:43.641282 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.641233    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-rcsnb" podStartSLOduration=4.11663349 podStartE2EDuration="21.641220293s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:50:24.946880529 +0000 UTC m=+3.073263674" lastFinishedPulling="2026-05-20 13:50:42.471467345 +0000 UTC m=+20.597850477" observedRunningTime="2026-05-20 13:50:43.64065647 +0000 UTC m=+21.767039615" watchObservedRunningTime="2026-05-20 13:50:43.641220293 +0000 UTC m=+21.767603445"
May 20 13:50:43.687797 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.687742    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-49mt5" podStartSLOduration=4.143793447 podStartE2EDuration="21.687725538s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:50:24.932176187 +0000 UTC m=+3.058559318" lastFinishedPulling="2026-05-20 13:50:42.476108261 +0000 UTC m=+20.602491409" observedRunningTime="2026-05-20 13:50:43.687315749 +0000 UTC m=+21.813698914" watchObservedRunningTime="2026-05-20 13:50:43.687725538 +0000 UTC m=+21.814108691"
May 20 13:50:43.704965 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:43.704899    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t2fbl" podStartSLOduration=4.246424329 podStartE2EDuration="21.70488133s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:50:24.934768175 +0000 UTC m=+3.061151306" lastFinishedPulling="2026-05-20 13:50:42.393225161 +0000 UTC m=+20.519608307" observedRunningTime="2026-05-20 13:50:43.704552493 +0000 UTC m=+21.830935651" watchObservedRunningTime="2026-05-20 13:50:43.70488133 +0000 UTC m=+21.831264502"
May 20 13:50:44.017044 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.017021    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:44.017196 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:44.017177    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:44.017267 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:44.017254    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret podName:e7efcced-743d-485e-a499-25e831588966 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:48.017233101 +0000 UTC m=+26.143616252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret") pod "global-pull-secret-syncer-jnqxh" (UID: "e7efcced-743d-485e-a499-25e831588966") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:44.059742 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.059710    2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 20 13:50:44.403993 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.403849    2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T13:50:44.059733057Z","UUID":"be291761-29ea-4497-8920-8d62b0e6708a","Handler":null,"Name":"","Endpoint":""}
May 20 13:50:44.405850 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.405818    2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 20 13:50:44.406007 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.405863    2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 20 13:50:44.463212 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.463175    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:44.463411 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:44.463318    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:44.604492 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.604406    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp" event={"ID":"b7b7bb27-c274-4fa1-a5fb-97fe855429c3","Type":"ContainerStarted","Data":"1b03660a34d2c69c8a0d2a023e811364278ce0fb2ce2152ff2c54fcc72df23c6"}
May 20 13:50:44.604492 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.604450    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp" event={"ID":"b7b7bb27-c274-4fa1-a5fb-97fe855429c3","Type":"ContainerStarted","Data":"440ed49e7778e61497291b05431f02af4e46e1a394ad3f431d9e71f3a93e85fc"}
May 20 13:50:44.604492 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.604464    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp" event={"ID":"b7b7bb27-c274-4fa1-a5fb-97fe855429c3","Type":"ContainerStarted","Data":"9178bc2d7d7447b4c807a06df5410b86b9e2aecabfd0f277363f290124c4623f"}
May 20 13:50:44.604492 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.604479    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp" event={"ID":"b7b7bb27-c274-4fa1-a5fb-97fe855429c3","Type":"ContainerStarted","Data":"26937a34f0d609fd60b4e0b293047741c969e395bc054474f9eb12beb4fefc8e"}
May 20 13:50:44.604492 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.604491    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp" event={"ID":"b7b7bb27-c274-4fa1-a5fb-97fe855429c3","Type":"ContainerStarted","Data":"4abaa8cc861fee0b1de5db2cd54294ecf7024f39e1cd0a5c76d99bf63820d057"}
May 20 13:50:44.606266 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.606236    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6" event={"ID":"f7dbfe28-779d-4c80-aa72-8e0cd8af4385","Type":"ContainerStarted","Data":"5d982378e5c20f1e45b4e7ed1d7a1879ed441be2ae8721777fe356cb2aa42965"}
May 20 13:50:44.607671 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.607641    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-th9rk" event={"ID":"148ba522-00f2-49f0-b1b2-0d3c8860b2eb","Type":"ContainerStarted","Data":"2d54cceeb413eeb4d3b4e0290edd55e3caf2889bf74fcc03b270231019e80363"}
May 20 13:50:44.623793 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:44.623735    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-th9rk" podStartSLOduration=5.102414734 podStartE2EDuration="22.623716811s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:50:24.94286951 +0000 UTC m=+3.069252645" lastFinishedPulling="2026-05-20 13:50:42.464171574 +0000 UTC m=+20.590554722" observedRunningTime="2026-05-20 13:50:44.623398715 +0000 UTC m=+22.749781863" watchObservedRunningTime="2026-05-20 13:50:44.623716811 +0000 UTC m=+22.750099963"
May 20 13:50:45.462508 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:45.462472    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:45.462670 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:45.462472    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:45.462670 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:45.462608    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:45.462797 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:45.462675    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jnqxh" podUID="e7efcced-743d-485e-a499-25e831588966"
May 20 13:50:45.468751 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:45.468722    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:45.469633 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:45.469396    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:45.612044 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:45.611998    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6" event={"ID":"f7dbfe28-779d-4c80-aa72-8e0cd8af4385","Type":"ContainerStarted","Data":"6f96d80920c49090c8c52597c6784aa90e588f3f7e169039cf6ca0d0005db7e9"}
May 20 13:50:45.612523 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:45.612478    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:45.612833 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:45.612808    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t2fbl"
May 20 13:50:45.632253 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:45.632195    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-shnw6" podStartSLOduration=3.332259273 podStartE2EDuration="23.63217467s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:50:24.932665388 +0000 UTC m=+3.059048519" lastFinishedPulling="2026-05-20 13:50:45.232580784 +0000 UTC m=+23.358963916" observedRunningTime="2026-05-20 13:50:45.631025214 +0000 UTC m=+23.757408368" watchObservedRunningTime="2026-05-20 13:50:45.63217467 +0000 UTC m=+23.758557824"
May 20 13:50:46.462418 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:46.462384    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:46.462608 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:46.462555    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:47.462274 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:47.462240    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:47.462274 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:47.462266    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:47.462896 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:47.462403    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:47.462896 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:47.462515    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jnqxh" podUID="e7efcced-743d-485e-a499-25e831588966"
May 20 13:50:48.049555 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:48.049523    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:48.049698 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:48.049620    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:48.049762 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:48.049666    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret podName:e7efcced-743d-485e-a499-25e831588966 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:56.049654019 +0000 UTC m=+34.176037150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret") pod "global-pull-secret-syncer-jnqxh" (UID: "e7efcced-743d-485e-a499-25e831588966") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:48.463060 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:48.462976    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:48.463597 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:48.463080    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:48.618780 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:48.618743    2550 generic.go:358] "Generic (PLEG): container finished" podID="6502c66b-dfda-470b-98f8-6ec092bcfec2" containerID="e7a5a5c95669252307e0b7a2f164d01144a058d34ff28d994e69fd2e1e594e8f" exitCode=0
May 20 13:50:48.618938 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:48.618818    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx4t2" event={"ID":"6502c66b-dfda-470b-98f8-6ec092bcfec2","Type":"ContainerDied","Data":"e7a5a5c95669252307e0b7a2f164d01144a058d34ff28d994e69fd2e1e594e8f"}
May 20 13:50:48.621804 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:48.621779    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp" event={"ID":"b7b7bb27-c274-4fa1-a5fb-97fe855429c3","Type":"ContainerStarted","Data":"d330cb7bda6f9a4d2f8a2e64fb5185dc6f64e66387c2b5f127e60aec682284bb"}
May 20 13:50:49.462148 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:49.462114    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:49.462322 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:49.462119    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:49.462322 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:49.462206    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jnqxh" podUID="e7efcced-743d-485e-a499-25e831588966"
May 20 13:50:49.462449 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:49.462328    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:49.625595 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:49.625509    2550 generic.go:358] "Generic (PLEG): container finished" podID="6502c66b-dfda-470b-98f8-6ec092bcfec2" containerID="b9060d743daf335c4789bdc3859e7b6dff6686bc329da6a45e07ba6ebeb5b45b" exitCode=0
May 20 13:50:49.625595 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:49.625565    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx4t2" event={"ID":"6502c66b-dfda-470b-98f8-6ec092bcfec2","Type":"ContainerDied","Data":"b9060d743daf335c4789bdc3859e7b6dff6686bc329da6a45e07ba6ebeb5b45b"}
May 20 13:50:50.462487 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:50.462268    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:50.462673 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:50.462559    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:50.629768 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:50.629735    2550 generic.go:358] "Generic (PLEG): container finished" podID="6502c66b-dfda-470b-98f8-6ec092bcfec2" containerID="67f7f3d3de6968731b3b28cf50c8f0d881e8ddb5eceaf126c19b67533da60524" exitCode=0
May 20 13:50:50.630219 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:50.629810    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx4t2" event={"ID":"6502c66b-dfda-470b-98f8-6ec092bcfec2","Type":"ContainerDied","Data":"67f7f3d3de6968731b3b28cf50c8f0d881e8ddb5eceaf126c19b67533da60524"}
May 20 13:50:50.632925 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:50.632903    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp" event={"ID":"b7b7bb27-c274-4fa1-a5fb-97fe855429c3","Type":"ContainerStarted","Data":"cc0000042a14877269add42af524d9fd1de61bddfefd5a7c4960b33e81ab5c0a"}
May 20 13:50:50.633279 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:50.633252    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:50.633371 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:50.633282    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:50.633371 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:50.633298    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:50.648527 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:50.648503    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:50.651812 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:50.651794    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:50:50.678899 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:50.678857    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp" podStartSLOduration=10.156276323 podStartE2EDuration="28.678842246s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:50:24.943673624 +0000 UTC m=+3.070056771" lastFinishedPulling="2026-05-20 13:50:43.466239551 +0000 UTC m=+21.592622694" observedRunningTime="2026-05-20 13:50:50.677706216 +0000 UTC m=+28.804089369" watchObservedRunningTime="2026-05-20 13:50:50.678842246 +0000 UTC m=+28.805225396"
May 20 13:50:51.462573 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:51.462541    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:51.462755 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:51.462541    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:51.462755 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:51.462681    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:51.462878 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:51.462771    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jnqxh" podUID="e7efcced-743d-485e-a499-25e831588966"
May 20 13:50:52.249636 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:52.249314    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rvjzj"]
May 20 13:50:52.249636 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:52.249474    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:52.249636 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:52.249593    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:52.252305 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:52.251944    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jnqxh"]
May 20 13:50:52.252305 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:52.252040    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:52.252305 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:52.252133    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jnqxh" podUID="e7efcced-743d-485e-a499-25e831588966"
May 20 13:50:52.252931 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:52.252907    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7hds7"]
May 20 13:50:52.253025 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:52.252993    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:52.253135 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:52.253112    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:53.462773 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:53.462571    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:53.463144 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:53.462865    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:54.465970 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:54.465943    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:54.465970 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:54.465958    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:54.466441 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:54.466060    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:54.466441 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:54.466202    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jnqxh" podUID="e7efcced-743d-485e-a499-25e831588966"
May 20 13:50:55.463066 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:55.463026    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:55.463398 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:55.463179    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:50:56.118372 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.118321    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:56.118824 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.118386    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:56.118824 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:56.118509    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:56.118824 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:56.118534    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 20 13:50:56.118824 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:56.118591    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs podName:a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be nodeName:}" failed. No retries permitted until 2026-05-20 13:51:28.118569283 +0000 UTC m=+66.244952418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs") pod "network-metrics-daemon-rvjzj" (UID: "a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 13:50:56.118824 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:56.118612    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret podName:e7efcced-743d-485e-a499-25e831588966 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:12.118601996 +0000 UTC m=+50.244985132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret") pod "global-pull-secret-syncer-jnqxh" (UID: "e7efcced-743d-485e-a499-25e831588966") : object "kube-system"/"original-pull-secret" not registered
May 20 13:50:56.218744 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.218713    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnv5\" (UniqueName: \"kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5\") pod \"network-check-target-7hds7\" (UID: \"8d2b9089-b3ec-4941-91cc-7c3cada6db01\") " pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:56.218903 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:56.218840    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 13:50:56.218903 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:56.218854    2550 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 13:50:56.218903 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:56.218863    2550 projected.go:194] Error preparing data for projected volume kube-api-access-xbnv5 for pod openshift-network-diagnostics/network-check-target-7hds7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:56.219024 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:56.218911    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5 podName:8d2b9089-b3ec-4941-91cc-7c3cada6db01 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:28.218898447 +0000 UTC m=+66.345281578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xbnv5" (UniqueName: "kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5") pod "network-check-target-7hds7" (UID: "8d2b9089-b3ec-4941-91cc-7c3cada6db01") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 13:50:56.462120 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.462048    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:56.462120 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.462082    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:56.462284 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:56.462158    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jnqxh" podUID="e7efcced-743d-485e-a499-25e831588966"
May 20 13:50:56.462320 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:56.462271    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7hds7" podUID="8d2b9089-b3ec-4941-91cc-7c3cada6db01"
May 20 13:50:56.689683 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.689653    2550 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-129-117.ec2.internal" event="NodeReady"
May 20 13:50:56.689838 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.689826    2550 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 20 13:50:56.747174 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.747146    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lxrtg"]
May 20 13:50:56.764853 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.764810    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2vfk7"]
May 20 13:50:56.765000 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.764946    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:56.768243 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.768217    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 20 13:50:56.768813 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.768781    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jxfgl\""
May 20 13:50:56.768921 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.768849    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 20 13:50:56.778541 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.778520    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lxrtg"]
May 20 13:50:56.778541 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.778542    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2vfk7"]
May 20 13:50:56.778679 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.778630    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:50:56.781409 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.781376    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 20 13:50:56.781409 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.781389    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 20 13:50:56.781839 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.781823    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 20 13:50:56.783788 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.783764    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vxwgr\""
May 20 13:50:56.924691 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.924655    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-config-volume\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:56.924691 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.924690    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:50:56.924926 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.924711    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9jwh\" (UniqueName: \"kubernetes.io/projected/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-kube-api-access-h9jwh\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:56.924926 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.924781    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44lpb\" (UniqueName: \"kubernetes.io/projected/374503b2-0a1d-41f6-9fc7-b81f43151c0e-kube-api-access-44lpb\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:50:56.924926 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.924823    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:56.924926 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:56.924846    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-tmp-dir\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:57.026105 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.026023    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-config-volume\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:57.026105 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.026055    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:50:57.026105 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.026075    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h9jwh\" (UniqueName: \"kubernetes.io/projected/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-kube-api-access-h9jwh\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:57.026405 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:57.026174    2550 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:57.026405 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.026210    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-44lpb\" (UniqueName: \"kubernetes.io/projected/374503b2-0a1d-41f6-9fc7-b81f43151c0e-kube-api-access-44lpb\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:50:57.026405 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:57.026255    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert podName:374503b2-0a1d-41f6-9fc7-b81f43151c0e nodeName:}" failed. No retries permitted until 2026-05-20 13:50:57.526234427 +0000 UTC m=+35.652617572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert") pod "ingress-canary-2vfk7" (UID: "374503b2-0a1d-41f6-9fc7-b81f43151c0e") : secret "canary-serving-cert" not found
May 20 13:50:57.026405 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.026275    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:57.026405 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.026314    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-tmp-dir\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:57.026691 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:57.026427    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:57.026691 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:57.026476    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls podName:e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:57.526461655 +0000 UTC m=+35.652844796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls") pod "dns-default-lxrtg" (UID: "e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6") : secret "dns-default-metrics-tls" not found
May 20 13:50:57.026691 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.026576    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-tmp-dir\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:57.026691 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.026664    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-config-volume\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:57.039140 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.039115    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44lpb\" (UniqueName: \"kubernetes.io/projected/374503b2-0a1d-41f6-9fc7-b81f43151c0e-kube-api-access-44lpb\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:50:57.039245 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.039120    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9jwh\" (UniqueName: \"kubernetes.io/projected/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-kube-api-access-h9jwh\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:57.463099 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.463015    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:50:57.466017 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.465997    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 13:50:57.466210 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.466195    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vwf8m\""
May 20 13:50:57.529277 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.529243    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:57.529425 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.529359    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:50:57.529425 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:57.529418    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:57.529510 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:57.529443    2550 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:57.529510 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:57.529487    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert podName:374503b2-0a1d-41f6-9fc7-b81f43151c0e nodeName:}" failed. No retries permitted until 2026-05-20 13:50:58.529467524 +0000 UTC m=+36.655850669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert") pod "ingress-canary-2vfk7" (UID: "374503b2-0a1d-41f6-9fc7-b81f43151c0e") : secret "canary-serving-cert" not found
May 20 13:50:57.529510 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:57.529503    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls podName:e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6 nodeName:}" failed. No retries permitted until 2026-05-20 13:50:58.529495624 +0000 UTC m=+36.655878770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls") pod "dns-default-lxrtg" (UID: "e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6") : secret "dns-default-metrics-tls" not found
May 20 13:50:57.648569 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.648539    2550 generic.go:358] "Generic (PLEG): container finished" podID="6502c66b-dfda-470b-98f8-6ec092bcfec2" containerID="344a2fa608c898f04b14884d67b5a62f42df2ec14c3c16962eb47dd866e4a313" exitCode=0
May 20 13:50:57.648738 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:57.648577    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx4t2" event={"ID":"6502c66b-dfda-470b-98f8-6ec092bcfec2","Type":"ContainerDied","Data":"344a2fa608c898f04b14884d67b5a62f42df2ec14c3c16962eb47dd866e4a313"}
May 20 13:50:58.465713 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:58.465688    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:50:58.466256 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:58.465688    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:50:58.468671 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:58.468649    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 20 13:50:58.469707 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:58.469688    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 13:50:58.469816 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:58.469690    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 13:50:58.469816 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:58.469690    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtwjf\""
May 20 13:50:58.536560 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:58.536524    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:50:58.536737 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:58.536582    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:50:58.536737 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:58.536674    2550 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:50:58.536737 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:58.536707    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:50:58.536737 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:58.536736    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert podName:374503b2-0a1d-41f6-9fc7-b81f43151c0e nodeName:}" failed. No retries permitted until 2026-05-20 13:51:00.536722163 +0000 UTC m=+38.663105300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert") pod "ingress-canary-2vfk7" (UID: "374503b2-0a1d-41f6-9fc7-b81f43151c0e") : secret "canary-serving-cert" not found
May 20 13:50:58.536916 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:50:58.536757    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls podName:e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:00.536742371 +0000 UTC m=+38.663125522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls") pod "dns-default-lxrtg" (UID: "e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6") : secret "dns-default-metrics-tls" not found
May 20 13:50:58.653646 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:58.653616    2550 generic.go:358] "Generic (PLEG): container finished" podID="6502c66b-dfda-470b-98f8-6ec092bcfec2" containerID="f4d5f5c8c6ad9694b861cae69bf4e2341574724fdb0d600ac2f8cac1186faa9a" exitCode=0
May 20 13:50:58.653794 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:58.653674    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx4t2" event={"ID":"6502c66b-dfda-470b-98f8-6ec092bcfec2","Type":"ContainerDied","Data":"f4d5f5c8c6ad9694b861cae69bf4e2341574724fdb0d600ac2f8cac1186faa9a"}
May 20 13:50:59.658067 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:59.657796    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qx4t2" event={"ID":"6502c66b-dfda-470b-98f8-6ec092bcfec2","Type":"ContainerStarted","Data":"765f88c88b5111514f857972c2d192229b9df079cd0684d21606c84c451f0574"}
May 20 13:50:59.681189 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:50:59.681135    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qx4t2" podStartSLOduration=6.081904658 podStartE2EDuration="37.681116909s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:50:24.938538392 +0000 UTC m=+3.064921525" lastFinishedPulling="2026-05-20 13:50:56.537750635 +0000 UTC m=+34.664133776" observedRunningTime="2026-05-20 13:50:59.679620963 +0000 UTC m=+37.806004115" watchObservedRunningTime="2026-05-20 13:50:59.681116909 +0000 UTC m=+37.807500065"
May 20 13:51:00.553961 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:00.553911    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:51:00.553961 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:00.553973    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:51:00.554183 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:00.554058    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:00.554183 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:00.554058    2550 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:00.554183 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:00.554107    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls podName:e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:04.554094071 +0000 UTC m=+42.680477202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls") pod "dns-default-lxrtg" (UID: "e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6") : secret "dns-default-metrics-tls" not found
May 20 13:51:00.554183 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:00.554120    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert podName:374503b2-0a1d-41f6-9fc7-b81f43151c0e nodeName:}" failed. No retries permitted until 2026-05-20 13:51:04.55411398 +0000 UTC m=+42.680497111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert") pod "ingress-canary-2vfk7" (UID: "374503b2-0a1d-41f6-9fc7-b81f43151c0e") : secret "canary-serving-cert" not found
May 20 13:51:04.583810 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:04.583769    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:51:04.583810 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:04.583819    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:51:04.584289 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:04.583909    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:04.584289 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:04.583937    2550 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:04.584289 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:04.583961    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls podName:e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:12.583949259 +0000 UTC m=+50.710332391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls") pod "dns-default-lxrtg" (UID: "e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6") : secret "dns-default-metrics-tls" not found
May 20 13:51:04.584289 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:04.584012    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert podName:374503b2-0a1d-41f6-9fc7-b81f43151c0e nodeName:}" failed. No retries permitted until 2026-05-20 13:51:12.583994743 +0000 UTC m=+50.710377891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert") pod "ingress-canary-2vfk7" (UID: "374503b2-0a1d-41f6-9fc7-b81f43151c0e") : secret "canary-serving-cert" not found
May 20 13:51:12.137482 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:12.137441    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:51:12.140300 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:12.140275    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e7efcced-743d-485e-a499-25e831588966-original-pull-secret\") pod \"global-pull-secret-syncer-jnqxh\" (UID: \"e7efcced-743d-485e-a499-25e831588966\") " pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:51:12.275697 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:12.275656    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jnqxh"
May 20 13:51:12.412677 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:12.412592    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jnqxh"]
May 20 13:51:12.415690 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:51:12.415661    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7efcced_743d_485e_a499_25e831588966.slice/crio-609981526bb04351952b0cc0b00aa887d77f7c45c51ee6bf9532f66e1862cdf8 WatchSource:0}: Error finding container 609981526bb04351952b0cc0b00aa887d77f7c45c51ee6bf9532f66e1862cdf8: Status 404 returned error can't find the container with id 609981526bb04351952b0cc0b00aa887d77f7c45c51ee6bf9532f66e1862cdf8
May 20 13:51:12.641989 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:12.641947    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:51:12.642242 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:12.642020    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:51:12.642242 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:12.642103    2550 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:12.642242 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:12.642111    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:12.642242 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:12.642155    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert podName:374503b2-0a1d-41f6-9fc7-b81f43151c0e nodeName:}" failed. No retries permitted until 2026-05-20 13:51:28.642141204 +0000 UTC m=+66.768524335 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert") pod "ingress-canary-2vfk7" (UID: "374503b2-0a1d-41f6-9fc7-b81f43151c0e") : secret "canary-serving-cert" not found
May 20 13:51:12.642242 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:12.642168    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls podName:e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6 nodeName:}" failed. No retries permitted until 2026-05-20 13:51:28.642161946 +0000 UTC m=+66.768545077 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls") pod "dns-default-lxrtg" (UID: "e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6") : secret "dns-default-metrics-tls" not found
May 20 13:51:12.686933 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:12.686840    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jnqxh" event={"ID":"e7efcced-743d-485e-a499-25e831588966","Type":"ContainerStarted","Data":"609981526bb04351952b0cc0b00aa887d77f7c45c51ee6bf9532f66e1862cdf8"}
May 20 13:51:16.695351 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:16.695253    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jnqxh" event={"ID":"e7efcced-743d-485e-a499-25e831588966","Type":"ContainerStarted","Data":"86a9d49993095b6c4f8d9cb8a2b7327bbd120fb24666bf1167ddf71c77751306"}
May 20 13:51:16.711992 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:16.711938    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jnqxh" podStartSLOduration=32.835641789 podStartE2EDuration="36.711924834s" podCreationTimestamp="2026-05-20 13:50:40 +0000 UTC" firstStartedPulling="2026-05-20 13:51:12.417389921 +0000 UTC m=+50.543773052" lastFinishedPulling="2026-05-20 13:51:16.293672953 +0000 UTC m=+54.420056097" observedRunningTime="2026-05-20 13:51:16.711634654 +0000 UTC m=+54.838017808" watchObservedRunningTime="2026-05-20 13:51:16.711924834 +0000 UTC m=+54.838307987"
May 20 13:51:22.647229 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:22.647199    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jbcdp"
May 20 13:51:27.785543 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.785509    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c"]
May 20 13:51:27.787849 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.787832    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c"
May 20 13:51:27.792017 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.791726    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 20 13:51:27.792017 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.791857    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 20 13:51:27.792017 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.791869    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 20 13:51:27.792017 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.791909    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6rwbw\""
May 20 13:51:27.792221 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.792112    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 20 13:51:27.798154 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.798128    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c"]
May 20 13:51:27.853949 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.853913    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75974a58-0197-4616-97a3-c68d3a944e51-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c\" (UID: \"75974a58-0197-4616-97a3-c68d3a944e51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c"
May 20 13:51:27.854118 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.853976    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2m2w\" (UniqueName: \"kubernetes.io/projected/75974a58-0197-4616-97a3-c68d3a944e51-kube-api-access-m2m2w\") pod \"managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c\" (UID: \"75974a58-0197-4616-97a3-c68d3a944e51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c"
May 20 13:51:27.955140 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.955110    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75974a58-0197-4616-97a3-c68d3a944e51-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c\" (UID: \"75974a58-0197-4616-97a3-c68d3a944e51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c"
May 20 13:51:27.955260 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.955174    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m2m2w\" (UniqueName: \"kubernetes.io/projected/75974a58-0197-4616-97a3-c68d3a944e51-kube-api-access-m2m2w\") pod \"managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c\" (UID: \"75974a58-0197-4616-97a3-c68d3a944e51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c"
May 20 13:51:27.957627 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.957608    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75974a58-0197-4616-97a3-c68d3a944e51-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c\" (UID: \"75974a58-0197-4616-97a3-c68d3a944e51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c"
May 20 13:51:27.964478 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:27.964445    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2m2w\" (UniqueName: \"kubernetes.io/projected/75974a58-0197-4616-97a3-c68d3a944e51-kube-api-access-m2m2w\") pod \"managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c\" (UID: \"75974a58-0197-4616-97a3-c68d3a944e51\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c"
May 20 13:51:28.108679 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.108570    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c"
May 20 13:51:28.157746 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.157712    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:51:28.160832 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.160805    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 13:51:28.168225 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:28.168199    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 20 13:51:28.168367 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:28.168304    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs podName:a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be nodeName:}" failed. No retries permitted until 2026-05-20 13:52:32.168281133 +0000 UTC m=+130.294664266 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs") pod "network-metrics-daemon-rvjzj" (UID: "a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be") : secret "metrics-daemon-secret" not found
May 20 13:51:28.230101 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.230066    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c"]
May 20 13:51:28.233350 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:51:28.233305    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75974a58_0197_4616_97a3_c68d3a944e51.slice/crio-60913ee232ad44b93a8c66edb8495fccf89e9658ac9b045a98602253e41ffaf0 WatchSource:0}: Error finding container 60913ee232ad44b93a8c66edb8495fccf89e9658ac9b045a98602253e41ffaf0: Status 404 returned error can't find the container with id 60913ee232ad44b93a8c66edb8495fccf89e9658ac9b045a98602253e41ffaf0
May 20 13:51:28.258344 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.258315    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnv5\" (UniqueName: \"kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5\") pod \"network-check-target-7hds7\" (UID: \"8d2b9089-b3ec-4941-91cc-7c3cada6db01\") " pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:51:28.261154 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.261130    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 13:51:28.273099 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.273077    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 13:51:28.282760 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.282737    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbnv5\" (UniqueName: \"kubernetes.io/projected/8d2b9089-b3ec-4941-91cc-7c3cada6db01-kube-api-access-xbnv5\") pod \"network-check-target-7hds7\" (UID: \"8d2b9089-b3ec-4941-91cc-7c3cada6db01\") " pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:51:28.483157 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.483077    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtwjf\""
May 20 13:51:28.490738 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.490715    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:51:28.625781 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.625751    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7hds7"]
May 20 13:51:28.628845 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:51:28.628817    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2b9089_b3ec_4941_91cc_7c3cada6db01.slice/crio-0545f0edeb37aa4007df06fc40a3f88536720db0320ec05b3ffaefc087613ec5 WatchSource:0}: Error finding container 0545f0edeb37aa4007df06fc40a3f88536720db0320ec05b3ffaefc087613ec5: Status 404 returned error can't find the container with id 0545f0edeb37aa4007df06fc40a3f88536720db0320ec05b3ffaefc087613ec5
May 20 13:51:28.662933 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.662898    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:51:28.663129 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.662967    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:51:28.663129 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:28.663063    2550 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:51:28.663129 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:28.663073    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:51:28.663129 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:28.663118    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert podName:374503b2-0a1d-41f6-9fc7-b81f43151c0e nodeName:}" failed. No retries permitted until 2026-05-20 13:52:00.6631047 +0000 UTC m=+98.789487831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert") pod "ingress-canary-2vfk7" (UID: "374503b2-0a1d-41f6-9fc7-b81f43151c0e") : secret "canary-serving-cert" not found
May 20 13:51:28.663331 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:51:28.663147    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls podName:e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:00.663128664 +0000 UTC m=+98.789511795 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls") pod "dns-default-lxrtg" (UID: "e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6") : secret "dns-default-metrics-tls" not found
May 20 13:51:28.718200 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.718161    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7hds7" event={"ID":"8d2b9089-b3ec-4941-91cc-7c3cada6db01","Type":"ContainerStarted","Data":"0545f0edeb37aa4007df06fc40a3f88536720db0320ec05b3ffaefc087613ec5"}
May 20 13:51:28.719090 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:28.719068    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c" event={"ID":"75974a58-0197-4616-97a3-c68d3a944e51","Type":"ContainerStarted","Data":"60913ee232ad44b93a8c66edb8495fccf89e9658ac9b045a98602253e41ffaf0"}
May 20 13:51:33.730096 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:33.730055    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c" event={"ID":"75974a58-0197-4616-97a3-c68d3a944e51","Type":"ContainerStarted","Data":"e7f6401b3afca6c0da3a8a19c0defdb4384b58d6cc219a018cfa10f4c2772da4"}
May 20 13:51:33.731377 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:33.731354    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7hds7" event={"ID":"8d2b9089-b3ec-4941-91cc-7c3cada6db01","Type":"ContainerStarted","Data":"33e09dab578ada710871b67695f4f8fd32f61f702a71c826e918dbdeee2a5f01"}
May 20 13:51:33.731526 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:33.731513    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:51:33.747761 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:33.747712    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff76bc5b-d9g9c" podStartSLOduration=2.318876425 podStartE2EDuration="6.747699407s" podCreationTimestamp="2026-05-20 13:51:27 +0000 UTC" firstStartedPulling="2026-05-20 13:51:28.235066581 +0000 UTC m=+66.361449711" lastFinishedPulling="2026-05-20 13:51:32.663889563 +0000 UTC m=+70.790272693" observedRunningTime="2026-05-20 13:51:33.746710294 +0000 UTC m=+71.873093448" watchObservedRunningTime="2026-05-20 13:51:33.747699407 +0000 UTC m=+71.874082560"
May 20 13:51:33.762178 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:51:33.762130    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7hds7" podStartSLOduration=67.737307963 podStartE2EDuration="1m11.762112358s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:51:28.630698099 +0000 UTC m=+66.757081229" lastFinishedPulling="2026-05-20 13:51:32.655502488 +0000 UTC m=+70.781885624" observedRunningTime="2026-05-20 13:51:33.761489317 +0000 UTC m=+71.887872471" watchObservedRunningTime="2026-05-20 13:51:33.762112358 +0000 UTC m=+71.888495520"
May 20 13:52:00.690422 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:00.690272    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:52:00.690422 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:00.690327    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:52:00.690896 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:00.690449    2550 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 20 13:52:00.690896 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:00.690514    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 20 13:52:00.690896 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:00.690528    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert podName:374503b2-0a1d-41f6-9fc7-b81f43151c0e nodeName:}" failed. No retries permitted until 2026-05-20 13:53:04.690510592 +0000 UTC m=+162.816893724 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert") pod "ingress-canary-2vfk7" (UID: "374503b2-0a1d-41f6-9fc7-b81f43151c0e") : secret "canary-serving-cert" not found
May 20 13:52:00.690896 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:00.690563    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls podName:e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:04.690550494 +0000 UTC m=+162.816933626 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls") pod "dns-default-lxrtg" (UID: "e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6") : secret "dns-default-metrics-tls" not found
May 20 13:52:04.736175 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:04.736148    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7hds7"
May 20 13:52:32.214532 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:32.214474    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:52:32.215053 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:32.214653    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 20 13:52:32.215053 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:32.214752    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs podName:a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be nodeName:}" failed. No retries permitted until 2026-05-20 13:54:34.214730847 +0000 UTC m=+252.341113993 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs") pod "network-metrics-daemon-rvjzj" (UID: "a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be") : secret "metrics-daemon-secret" not found
May 20 13:52:45.499356 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.499303    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-nddf5"]
May 20 13:52:45.501060 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.501038    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"]
May 20 13:52:45.501203 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.501185    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-nddf5"
May 20 13:52:45.502502 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.502485    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-656dc888b4-j7s2n"]
May 20 13:52:45.502665 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.502646    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:52:45.504482 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.504462    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.505148 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.505123    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lb6kz\""
May 20 13:52:45.505772 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.505748    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 20 13:52:45.505923 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.505796    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 20 13:52:45.508058 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.506192    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 20 13:52:45.508058 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.504553    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 20 13:52:45.508058 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.506900    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls\") pod \"cluster-samples-operator-697587599-sklwb\" (UID: \"d7690d16-8a38-45f9-a36e-da5f8fc16332\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:52:45.508058 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.507048    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-m6wrt\""
May 20 13:52:45.508058 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.507109    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vgz\" (UniqueName: \"kubernetes.io/projected/4baead3c-57b6-4b3d-966b-daf23038fa89-kube-api-access-55vgz\") pod \"volume-data-source-validator-5cf8c89d64-nddf5\" (UID: \"4baead3c-57b6-4b3d-966b-daf23038fa89\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-nddf5"
May 20 13:52:45.508058 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.506198    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 20 13:52:45.508058 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.507227    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfwkn\" (UniqueName: \"kubernetes.io/projected/d7690d16-8a38-45f9-a36e-da5f8fc16332-kube-api-access-wfwkn\") pod \"cluster-samples-operator-697587599-sklwb\" (UID: \"d7690d16-8a38-45f9-a36e-da5f8fc16332\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:52:45.508789 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.508771    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 20 13:52:45.508866 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.508790    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 20 13:52:45.509317 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.509297    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 20 13:52:45.509821 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.509806    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 20 13:52:45.510537 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.510520    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c24c4\""
May 20 13:52:45.510784 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.510765    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 20 13:52:45.510866 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.510774    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 20 13:52:45.513907 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.513884    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-nddf5"]
May 20 13:52:45.518234 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.518000    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"]
May 20 13:52:45.522412 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.522386    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-656dc888b4-j7s2n"]
May 20 13:52:45.598583 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.598551    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"]
May 20 13:52:45.600415 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.600398    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:45.603176 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.603149    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 20 13:52:45.603459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.603442    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rqfq4\""
May 20 13:52:45.603536 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.603505    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 20 13:52:45.603595 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.603571    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 20 13:52:45.604867 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.604853    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 20 13:52:45.608649 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.608628    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:45.608730 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.608667    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:45.608780 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.608734    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mkpd\" (UniqueName: \"kubernetes.io/projected/daad92a8-291c-4a57-9bb3-40d58be05740-kube-api-access-5mkpd\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.608780 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.608774    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls\") pod \"cluster-samples-operator-697587599-sklwb\" (UID: \"d7690d16-8a38-45f9-a36e-da5f8fc16332\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:52:45.608877 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.608794    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-55vgz\" (UniqueName: \"kubernetes.io/projected/4baead3c-57b6-4b3d-966b-daf23038fa89-kube-api-access-55vgz\") pod \"volume-data-source-validator-5cf8c89d64-nddf5\" (UID: \"4baead3c-57b6-4b3d-966b-daf23038fa89\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-nddf5"
May 20 13:52:45.608877 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.608811    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.608877 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.608867    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wfwkn\" (UniqueName: \"kubernetes.io/projected/d7690d16-8a38-45f9-a36e-da5f8fc16332-kube-api-access-wfwkn\") pod \"cluster-samples-operator-697587599-sklwb\" (UID: \"d7690d16-8a38-45f9-a36e-da5f8fc16332\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:52:45.608999 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:45.608900    2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 13:52:45.608999 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:45.608956    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls podName:d7690d16-8a38-45f9-a36e-da5f8fc16332 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:46.108936634 +0000 UTC m=+144.235319784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls") pod "cluster-samples-operator-697587599-sklwb" (UID: "d7690d16-8a38-45f9-a36e-da5f8fc16332") : secret "samples-operator-tls" not found
May 20 13:52:45.609101 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.609002    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-default-certificate\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.609101 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.609037    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.609101 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.609085    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5zc\" (UniqueName: \"kubernetes.io/projected/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-kube-api-access-2n5zc\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:45.609220 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.609118    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-stats-auth\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.614961 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.614939    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"]
May 20 13:52:45.626187 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.626163    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vgz\" (UniqueName: \"kubernetes.io/projected/4baead3c-57b6-4b3d-966b-daf23038fa89-kube-api-access-55vgz\") pod \"volume-data-source-validator-5cf8c89d64-nddf5\" (UID: \"4baead3c-57b6-4b3d-966b-daf23038fa89\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-nddf5"
May 20 13:52:45.626559 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.626537    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfwkn\" (UniqueName: \"kubernetes.io/projected/d7690d16-8a38-45f9-a36e-da5f8fc16332-kube-api-access-wfwkn\") pod \"cluster-samples-operator-697587599-sklwb\" (UID: \"d7690d16-8a38-45f9-a36e-da5f8fc16332\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:52:45.709768 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.709718    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-stats-auth\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.709768 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.709772    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:45.710044 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.709792    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:45.710044 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.709831    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5mkpd\" (UniqueName: \"kubernetes.io/projected/daad92a8-291c-4a57-9bb3-40d58be05740-kube-api-access-5mkpd\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.710044 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.709889    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.710044 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:45.709906    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 13:52:45.710044 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:45.709970    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 20 13:52:45.710044 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:45.710019    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls podName:74f1e9a4-b0ef-48e5-8107-1b9a218414e1 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:46.209999096 +0000 UTC m=+144.336382230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-2xrcl" (UID: "74f1e9a4-b0ef-48e5-8107-1b9a218414e1") : secret "cluster-monitoring-operator-tls" not found
May 20 13:52:45.710044 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:45.710038    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs podName:daad92a8-291c-4a57-9bb3-40d58be05740 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:46.210027754 +0000 UTC m=+144.336410886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs") pod "router-default-656dc888b4-j7s2n" (UID: "daad92a8-291c-4a57-9bb3-40d58be05740") : secret "router-metrics-certs-default" not found
May 20 13:52:45.710044 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.710008    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-default-certificate\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.710399 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.710085    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.710399 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.710120    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5zc\" (UniqueName: \"kubernetes.io/projected/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-kube-api-access-2n5zc\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:45.710399 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:45.710271    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle podName:daad92a8-291c-4a57-9bb3-40d58be05740 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:46.210249444 +0000 UTC m=+144.336632589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle") pod "router-default-656dc888b4-j7s2n" (UID: "daad92a8-291c-4a57-9bb3-40d58be05740") : configmap references non-existent config key: service-ca.crt
May 20 13:52:45.710620 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.710601    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:45.712282 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.712265    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-default-certificate\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.712360 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.712306    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-stats-auth\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.718929 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.718903    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5zc\" (UniqueName: \"kubernetes.io/projected/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-kube-api-access-2n5zc\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:45.719053 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.719028    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mkpd\" (UniqueName: \"kubernetes.io/projected/daad92a8-291c-4a57-9bb3-40d58be05740-kube-api-access-5mkpd\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:45.813511 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.813407    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-nddf5"
May 20 13:52:45.844263 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.844231    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cb7476d84-xwppj"]
May 20 13:52:45.846852 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.846831    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:45.849921 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.849683    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 20 13:52:45.849921 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.849703    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 20 13:52:45.851074 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.850827    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 20 13:52:45.851074 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.850959    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gr7x7\""
May 20 13:52:45.855012 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.854994    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 20 13:52:45.858086 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.858061    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cb7476d84-xwppj"]
May 20 13:52:45.912876 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.912506    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-trusted-ca\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:45.912876 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.912679    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-installation-pull-secrets\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:45.912876 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.912729    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-registry-certificates\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:45.912876 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.912780    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8pcj\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-kube-api-access-q8pcj\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:45.912876 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.912819    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c03f37cb-a242-4358-966d-af487e7d58b9-ca-trust-extracted\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:45.912876 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.912874    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-bound-sa-token\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:45.913388 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.912912    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:45.913388 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.912962    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-image-registry-private-configuration\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:45.941112 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:45.941079    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-nddf5"]
May 20 13:52:45.944395 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:52:45.944365    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4baead3c_57b6_4b3d_966b_daf23038fa89.slice/crio-3f196e50a79b4cc6dcb80a51ef5713214e6ca21c49da14f21229718724e81f1e WatchSource:0}: Error finding container 3f196e50a79b4cc6dcb80a51ef5713214e6ca21c49da14f21229718724e81f1e: Status 404 returned error can't find the container with id 3f196e50a79b4cc6dcb80a51ef5713214e6ca21c49da14f21229718724e81f1e
May 20 13:52:46.013551 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.013514    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.013737 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.013569    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-image-registry-private-configuration\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.013737 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.013603    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-trusted-ca\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.013737 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.013663    2550 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:52:46.013737 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.013687    2550 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb7476d84-xwppj: secret "image-registry-tls" not found
May 20 13:52:46.013737 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.013741    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls podName:c03f37cb-a242-4358-966d-af487e7d58b9 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:46.513724836 +0000 UTC m=+144.640107978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls") pod "image-registry-6cb7476d84-xwppj" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9") : secret "image-registry-tls" not found
May 20 13:52:46.014013 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.013667    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-installation-pull-secrets\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.014013 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.013830    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-registry-certificates\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.014013 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.013890    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q8pcj\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-kube-api-access-q8pcj\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.014013 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.013917    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c03f37cb-a242-4358-966d-af487e7d58b9-ca-trust-extracted\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.014013 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.013966    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-bound-sa-token\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.014312 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.014289    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c03f37cb-a242-4358-966d-af487e7d58b9-ca-trust-extracted\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.014474 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.014456    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-registry-certificates\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.014700 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.014678    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-trusted-ca\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.016613 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.016595    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-image-registry-private-configuration\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.016667 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.016641    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-installation-pull-secrets\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.022625 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.022603    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-bound-sa-token\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.022718 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.022647    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8pcj\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-kube-api-access-q8pcj\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.115053 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.114957    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls\") pod \"cluster-samples-operator-697587599-sklwb\" (UID: \"d7690d16-8a38-45f9-a36e-da5f8fc16332\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:52:46.115232 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.115128    2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 13:52:46.115232 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.115224    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls podName:d7690d16-8a38-45f9-a36e-da5f8fc16332 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:47.11520284 +0000 UTC m=+145.241585985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls") pod "cluster-samples-operator-697587599-sklwb" (UID: "d7690d16-8a38-45f9-a36e-da5f8fc16332") : secret "samples-operator-tls" not found
May 20 13:52:46.216060 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.216026    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:46.216210 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.216087    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:46.216210 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.216119    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:46.216371 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.216215    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 20 13:52:46.216371 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.216247    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 13:52:46.216371 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.216256    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs podName:daad92a8-291c-4a57-9bb3-40d58be05740 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:47.216242573 +0000 UTC m=+145.342625704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs") pod "router-default-656dc888b4-j7s2n" (UID: "daad92a8-291c-4a57-9bb3-40d58be05740") : secret "router-metrics-certs-default" not found
May 20 13:52:46.216371 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.216323    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle podName:daad92a8-291c-4a57-9bb3-40d58be05740 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:47.216306707 +0000 UTC m=+145.342689856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle") pod "router-default-656dc888b4-j7s2n" (UID: "daad92a8-291c-4a57-9bb3-40d58be05740") : configmap references non-existent config key: service-ca.crt
May 20 13:52:46.216371 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.216359    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls podName:74f1e9a4-b0ef-48e5-8107-1b9a218414e1 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:47.21632934 +0000 UTC m=+145.342712475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-2xrcl" (UID: "74f1e9a4-b0ef-48e5-8107-1b9a218414e1") : secret "cluster-monitoring-operator-tls" not found
May 20 13:52:46.518867 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.518826    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:46.519368 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.519010    2550 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:52:46.519368 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.519040    2550 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb7476d84-xwppj: secret "image-registry-tls" not found
May 20 13:52:46.519368 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:46.519114    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls podName:c03f37cb-a242-4358-966d-af487e7d58b9 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:47.519090548 +0000 UTC m=+145.645473693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls") pod "image-registry-6cb7476d84-xwppj" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9") : secret "image-registry-tls" not found
May 20 13:52:46.871214 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:46.871137    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-nddf5" event={"ID":"4baead3c-57b6-4b3d-966b-daf23038fa89","Type":"ContainerStarted","Data":"3f196e50a79b4cc6dcb80a51ef5713214e6ca21c49da14f21229718724e81f1e"}
May 20 13:52:47.125099 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:47.124998    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls\") pod \"cluster-samples-operator-697587599-sklwb\" (UID: \"d7690d16-8a38-45f9-a36e-da5f8fc16332\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:52:47.125276 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:47.125182    2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 13:52:47.125276 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:47.125270    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls podName:d7690d16-8a38-45f9-a36e-da5f8fc16332 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:49.125246006 +0000 UTC m=+147.251629152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls") pod "cluster-samples-operator-697587599-sklwb" (UID: "d7690d16-8a38-45f9-a36e-da5f8fc16332") : secret "samples-operator-tls" not found
May 20 13:52:47.225586 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:47.225563    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:47.225685 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:47.225608    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:47.225729 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:47.225695    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 13:52:47.225763 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:47.225729    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:47.225763 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:47.225747    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls podName:74f1e9a4-b0ef-48e5-8107-1b9a218414e1 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:49.225731351 +0000 UTC m=+147.352114501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-2xrcl" (UID: "74f1e9a4-b0ef-48e5-8107-1b9a218414e1") : secret "cluster-monitoring-operator-tls" not found
May 20 13:52:47.225763 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:47.225704    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 20 13:52:47.225872 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:47.225809    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs podName:daad92a8-291c-4a57-9bb3-40d58be05740 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:49.225790757 +0000 UTC m=+147.352173905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs") pod "router-default-656dc888b4-j7s2n" (UID: "daad92a8-291c-4a57-9bb3-40d58be05740") : secret "router-metrics-certs-default" not found
May 20 13:52:47.225872 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:47.225826    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle podName:daad92a8-291c-4a57-9bb3-40d58be05740 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:49.225818142 +0000 UTC m=+147.352201273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle") pod "router-default-656dc888b4-j7s2n" (UID: "daad92a8-291c-4a57-9bb3-40d58be05740") : configmap references non-existent config key: service-ca.crt
May 20 13:52:47.528617 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:47.528585    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:47.529003 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:47.528693    2550 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:52:47.529003 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:47.528704    2550 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb7476d84-xwppj: secret "image-registry-tls" not found
May 20 13:52:47.529003 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:47.528751    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls podName:c03f37cb-a242-4358-966d-af487e7d58b9 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:49.528737859 +0000 UTC m=+147.655120991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls") pod "image-registry-6cb7476d84-xwppj" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9") : secret "image-registry-tls" not found
May 20 13:52:47.874091 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:47.874011    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-nddf5" event={"ID":"4baead3c-57b6-4b3d-966b-daf23038fa89","Type":"ContainerStarted","Data":"fc60f9589917e25ae4acfaf734b4d9f903610b933b5118a9a36aa865f4e25a79"}
May 20 13:52:47.892278 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:47.892234    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-nddf5" podStartSLOduration=1.671422414 podStartE2EDuration="2.892222116s" podCreationTimestamp="2026-05-20 13:52:45 +0000 UTC" firstStartedPulling="2026-05-20 13:52:45.946283071 +0000 UTC m=+144.072666202" lastFinishedPulling="2026-05-20 13:52:47.167082771 +0000 UTC m=+145.293465904" observedRunningTime="2026-05-20 13:52:47.891050845 +0000 UTC m=+146.017433999" watchObservedRunningTime="2026-05-20 13:52:47.892222116 +0000 UTC m=+146.018605318"
May 20 13:52:49.141280 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:49.141243    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls\") pod \"cluster-samples-operator-697587599-sklwb\" (UID: \"d7690d16-8a38-45f9-a36e-da5f8fc16332\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:52:49.141736 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:49.141412    2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 13:52:49.141736 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:49.141476    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls podName:d7690d16-8a38-45f9-a36e-da5f8fc16332 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:53.141461533 +0000 UTC m=+151.267844664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls") pod "cluster-samples-operator-697587599-sklwb" (UID: "d7690d16-8a38-45f9-a36e-da5f8fc16332") : secret "samples-operator-tls" not found
May 20 13:52:49.242424 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:49.242385    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:49.242569 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:49.242471    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:49.242569 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:49.242517    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:49.242569 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:49.242535    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 20 13:52:49.242670 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:49.242598    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs podName:daad92a8-291c-4a57-9bb3-40d58be05740 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:53.242583995 +0000 UTC m=+151.368967127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs") pod "router-default-656dc888b4-j7s2n" (UID: "daad92a8-291c-4a57-9bb3-40d58be05740") : secret "router-metrics-certs-default" not found
May 20 13:52:49.242670 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:49.242618    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 13:52:49.242670 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:49.242624    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle podName:daad92a8-291c-4a57-9bb3-40d58be05740 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:53.242608546 +0000 UTC m=+151.368991691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle") pod "router-default-656dc888b4-j7s2n" (UID: "daad92a8-291c-4a57-9bb3-40d58be05740") : configmap references non-existent config key: service-ca.crt
May 20 13:52:49.242670 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:49.242655    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls podName:74f1e9a4-b0ef-48e5-8107-1b9a218414e1 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:53.242644918 +0000 UTC m=+151.369028069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-2xrcl" (UID: "74f1e9a4-b0ef-48e5-8107-1b9a218414e1") : secret "cluster-monitoring-operator-tls" not found
May 20 13:52:49.545449 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:49.545409    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:49.545622 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:49.545558    2550 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:52:49.545622 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:49.545580    2550 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb7476d84-xwppj: secret "image-registry-tls" not found
May 20 13:52:49.545695 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:49.545635    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls podName:c03f37cb-a242-4358-966d-af487e7d58b9 nodeName:}" failed. No retries permitted until 2026-05-20 13:52:53.545619721 +0000 UTC m=+151.672002852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls") pod "image-registry-6cb7476d84-xwppj" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9") : secret "image-registry-tls" not found
May 20 13:52:53.177482 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:53.177443    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls\") pod \"cluster-samples-operator-697587599-sklwb\" (UID: \"d7690d16-8a38-45f9-a36e-da5f8fc16332\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:52:53.177890 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:53.177581    2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 20 13:52:53.177890 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:53.177642    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls podName:d7690d16-8a38-45f9-a36e-da5f8fc16332 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:01.177627903 +0000 UTC m=+159.304011040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls") pod "cluster-samples-operator-697587599-sklwb" (UID: "d7690d16-8a38-45f9-a36e-da5f8fc16332") : secret "samples-operator-tls" not found
May 20 13:52:53.278083 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:53.278042    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:53.278254 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:53.278131    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:52:53.278254 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:53.278177    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:52:53.278254 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:53.278200    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle podName:daad92a8-291c-4a57-9bb3-40d58be05740 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:01.278181838 +0000 UTC m=+159.404564973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle") pod "router-default-656dc888b4-j7s2n" (UID: "daad92a8-291c-4a57-9bb3-40d58be05740") : configmap references non-existent config key: service-ca.crt
May 20 13:52:53.278455 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:53.278285    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 13:52:53.278455 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:53.278374    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls podName:74f1e9a4-b0ef-48e5-8107-1b9a218414e1 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:01.278354108 +0000 UTC m=+159.404737275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-2xrcl" (UID: "74f1e9a4-b0ef-48e5-8107-1b9a218414e1") : secret "cluster-monitoring-operator-tls" not found
May 20 13:52:53.278455 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:53.278399    2550 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 20 13:52:53.278455 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:53.278455    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs podName:daad92a8-291c-4a57-9bb3-40d58be05740 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:01.278440136 +0000 UTC m=+159.404823273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs") pod "router-default-656dc888b4-j7s2n" (UID: "daad92a8-291c-4a57-9bb3-40d58be05740") : secret "router-metrics-certs-default" not found
May 20 13:52:53.580003 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:53.579959    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:52:53.580180 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:53.580112    2550 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 20 13:52:53.580180 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:53.580134    2550 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cb7476d84-xwppj: secret "image-registry-tls" not found
May 20 13:52:53.580262 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:53.580189    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls podName:c03f37cb-a242-4358-966d-af487e7d58b9 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:01.580174627 +0000 UTC m=+159.706557758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls") pod "image-registry-6cb7476d84-xwppj" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9") : secret "image-registry-tls" not found
May 20 13:52:54.546781 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:54.546752    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rcsnb_576d8a0a-be91-4189-b61a-3cec8a8834f2/dns-node-resolver/0.log"
May 20 13:52:55.148035 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:55.148010    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2gc7l_27731076-9ec4-4347-8b2f-0baf5b1e8622/node-ca/0.log"
May 20 13:52:59.775272 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:59.775229    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-lxrtg" podUID="e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6"
May 20 13:52:59.786443 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:52:59.786408    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-2vfk7" podUID="374503b2-0a1d-41f6-9fc7-b81f43151c0e"
May 20 13:52:59.900801 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:52:59.900771    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lxrtg"
May 20 13:53:00.472045 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:00.472007    2550 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-rvjzj" podUID="a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be"
May 20 13:53:01.250910 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.250872    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls\") pod \"cluster-samples-operator-697587599-sklwb\" (UID: \"d7690d16-8a38-45f9-a36e-da5f8fc16332\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:53:01.253403 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.253379    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7690d16-8a38-45f9-a36e-da5f8fc16332-samples-operator-tls\") pod \"cluster-samples-operator-697587599-sklwb\" (UID: \"d7690d16-8a38-45f9-a36e-da5f8fc16332\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:53:01.351589 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.351549    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:53:01.351744 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.351608    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:53:01.351744 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.351638    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:53:01.351744 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:01.351732    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle podName:daad92a8-291c-4a57-9bb3-40d58be05740 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:17.351711868 +0000 UTC m=+175.478095003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle") pod "router-default-656dc888b4-j7s2n" (UID: "daad92a8-291c-4a57-9bb3-40d58be05740") : configmap references non-existent config key: service-ca.crt
May 20 13:53:01.351896 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:01.351752    2550 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 20 13:53:01.351896 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:01.351810    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls podName:74f1e9a4-b0ef-48e5-8107-1b9a218414e1 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:17.351792576 +0000 UTC m=+175.478175725 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-2xrcl" (UID: "74f1e9a4-b0ef-48e5-8107-1b9a218414e1") : secret "cluster-monitoring-operator-tls" not found
May 20 13:53:01.353979 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.353956    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daad92a8-291c-4a57-9bb3-40d58be05740-metrics-certs\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:53:01.420672 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.420633    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"
May 20 13:53:01.542072 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.541990    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb"]
May 20 13:53:01.653965 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.653918    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:53:01.656261 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.656241    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls\") pod \"image-registry-6cb7476d84-xwppj\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") " pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:53:01.757823 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.757781    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:53:01.886041 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.886000    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cb7476d84-xwppj"]
May 20 13:53:01.889499 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:01.889472    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03f37cb_a242_4358_966d_af487e7d58b9.slice/crio-cfc670682e4aa66c6637d4405e362f2dd4ed141ae714dc4ce1aa0e9866ddcbcc WatchSource:0}: Error finding container cfc670682e4aa66c6637d4405e362f2dd4ed141ae714dc4ce1aa0e9866ddcbcc: Status 404 returned error can't find the container with id cfc670682e4aa66c6637d4405e362f2dd4ed141ae714dc4ce1aa0e9866ddcbcc
May 20 13:53:01.905064 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.905029    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb" event={"ID":"d7690d16-8a38-45f9-a36e-da5f8fc16332","Type":"ContainerStarted","Data":"b436ae4c09befa727156b971b1fa8b231a18c14299188522c0e38ad84a12bbae"}
May 20 13:53:01.906164 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:01.906124    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj" event={"ID":"c03f37cb-a242-4358-966d-af487e7d58b9","Type":"ContainerStarted","Data":"cfc670682e4aa66c6637d4405e362f2dd4ed141ae714dc4ce1aa0e9866ddcbcc"}
May 20 13:53:02.910041 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:02.909999    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj" event={"ID":"c03f37cb-a242-4358-966d-af487e7d58b9","Type":"ContainerStarted","Data":"373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2"}
May 20 13:53:02.910527 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:02.910203    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:53:02.935570 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:02.935505    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj" podStartSLOduration=17.935484283 podStartE2EDuration="17.935484283s" podCreationTimestamp="2026-05-20 13:52:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:53:02.933836581 +0000 UTC m=+161.060219735" watchObservedRunningTime="2026-05-20 13:53:02.935484283 +0000 UTC m=+161.061867437"
May 20 13:53:03.914127 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:03.914087    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb" event={"ID":"d7690d16-8a38-45f9-a36e-da5f8fc16332","Type":"ContainerStarted","Data":"1b90d3b9f51be905e4b9e4de4bfb3cdbe440fc5fa9773c78fdee898720f33e20"}
May 20 13:53:03.914602 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:03.914134    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb" event={"ID":"d7690d16-8a38-45f9-a36e-da5f8fc16332","Type":"ContainerStarted","Data":"da42c77e366309b9fa9ce2986b894c8c5fbffa5dfb51ffac3b8b7f76f8d219a5"}
May 20 13:53:03.938244 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:03.938194    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-sklwb" podStartSLOduration=17.456951032 podStartE2EDuration="18.938179588s" podCreationTimestamp="2026-05-20 13:52:45 +0000 UTC" firstStartedPulling="2026-05-20 13:53:01.579619255 +0000 UTC m=+159.706002386" lastFinishedPulling="2026-05-20 13:53:03.06084781 +0000 UTC m=+161.187230942" observedRunningTime="2026-05-20 13:53:03.936552955 +0000 UTC m=+162.062936107" watchObservedRunningTime="2026-05-20 13:53:03.938179588 +0000 UTC m=+162.064562741"
May 20 13:53:04.780383 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:04.780325    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:53:04.780590 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:04.780395    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:53:04.782645 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:04.782620    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6-metrics-tls\") pod \"dns-default-lxrtg\" (UID: \"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6\") " pod="openshift-dns/dns-default-lxrtg"
May 20 13:53:04.782761 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:04.782744    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/374503b2-0a1d-41f6-9fc7-b81f43151c0e-cert\") pod \"ingress-canary-2vfk7\" (UID: \"374503b2-0a1d-41f6-9fc7-b81f43151c0e\") " pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:53:05.004188 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:05.004157    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jxfgl\""
May 20 13:53:05.012498 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:05.012474    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lxrtg"
May 20 13:53:05.135238 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:05.135207    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lxrtg"]
May 20 13:53:05.138091 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:05.138065    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode459bac0_e5e7_41b0_b9c4_6d6e49d6cfe6.slice/crio-8fce74c6053266ca5b8e8e1609de42f99fb4c0f8490737052059de36302bc514 WatchSource:0}: Error finding container 8fce74c6053266ca5b8e8e1609de42f99fb4c0f8490737052059de36302bc514: Status 404 returned error can't find the container with id 8fce74c6053266ca5b8e8e1609de42f99fb4c0f8490737052059de36302bc514
May 20 13:53:05.923240 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:05.923200    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lxrtg" event={"ID":"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6","Type":"ContainerStarted","Data":"8fce74c6053266ca5b8e8e1609de42f99fb4c0f8490737052059de36302bc514"}
May 20 13:53:06.926738 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:06.926701    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lxrtg" event={"ID":"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6","Type":"ContainerStarted","Data":"8672d88590ff74786407ab4a70e207264d86c0985773e3ca2c55a1aa3370eabc"}
May 20 13:53:06.926738 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:06.926737    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lxrtg" event={"ID":"e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6","Type":"ContainerStarted","Data":"9bcb688654def56bce12dae4db6578319f635495306078732b684eca577392aa"}
May 20 13:53:06.927251 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:06.926850    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lxrtg"
May 20 13:53:06.945224 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:06.945170    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-lxrtg" podStartSLOduration=129.740713107 podStartE2EDuration="2m10.945156867s" podCreationTimestamp="2026-05-20 13:50:56 +0000 UTC" firstStartedPulling="2026-05-20 13:53:05.140019503 +0000 UTC m=+163.266402634" lastFinishedPulling="2026-05-20 13:53:06.344463254 +0000 UTC m=+164.470846394" observedRunningTime="2026-05-20 13:53:06.94423445 +0000 UTC m=+165.070617603" watchObservedRunningTime="2026-05-20 13:53:06.945156867 +0000 UTC m=+165.071540019"
May 20 13:53:12.463440 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:12.463355    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:53:12.466567 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:12.466544    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vxwgr\""
May 20 13:53:12.474106 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:12.474078    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2vfk7"
May 20 13:53:12.611152 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:12.611118    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2vfk7"]
May 20 13:53:12.614063 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:12.614034    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374503b2_0a1d_41f6_9fc7_b81f43151c0e.slice/crio-89267b0efe4da3fd54bffa5514e38d9b7aecd3ece665c07a0c1a7d279f868751 WatchSource:0}: Error finding container 89267b0efe4da3fd54bffa5514e38d9b7aecd3ece665c07a0c1a7d279f868751: Status 404 returned error can't find the container with id 89267b0efe4da3fd54bffa5514e38d9b7aecd3ece665c07a0c1a7d279f868751
May 20 13:53:12.942214 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:12.942174    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2vfk7" event={"ID":"374503b2-0a1d-41f6-9fc7-b81f43151c0e","Type":"ContainerStarted","Data":"89267b0efe4da3fd54bffa5514e38d9b7aecd3ece665c07a0c1a7d279f868751"}
May 20 13:53:14.462355 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:14.462241    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:53:14.948617 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:14.948581    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2vfk7" event={"ID":"374503b2-0a1d-41f6-9fc7-b81f43151c0e","Type":"ContainerStarted","Data":"4386d233ebda163eef55a232a389e5b54312783a761d3c0755c30363da503b90"}
May 20 13:53:14.965136 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:14.965090    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2vfk7" podStartSLOduration=137.563794697 podStartE2EDuration="2m18.96507682s" podCreationTimestamp="2026-05-20 13:50:56 +0000 UTC" firstStartedPulling="2026-05-20 13:53:12.615863036 +0000 UTC m=+170.742246168" lastFinishedPulling="2026-05-20 13:53:14.017144942 +0000 UTC m=+172.143528291" observedRunningTime="2026-05-20 13:53:14.964805857 +0000 UTC m=+173.091189013" watchObservedRunningTime="2026-05-20 13:53:14.96507682 +0000 UTC m=+173.091460014"
May 20 13:53:16.931403 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:16.931372    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lxrtg"
May 20 13:53:17.068309 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.068271    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-gg6jn"]
May 20 13:53:17.070249 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.070228    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-gg6jn"
May 20 13:53:17.074790 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.074765    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4w9df\""
May 20 13:53:17.074916 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.074815    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 20 13:53:17.074916 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.074830    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 20 13:53:17.088184 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.088159    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-gg6jn"]
May 20 13:53:17.093978 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.093959    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cb7476d84-xwppj"]
May 20 13:53:17.098793 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.098767    2550 patch_prober.go:28] interesting pod/image-registry-6cb7476d84-xwppj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 20 13:53:17.098908 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.098827    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj" podUID="c03f37cb-a242-4358-966d-af487e7d58b9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 13:53:17.174490 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.174422    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksl9g\" (UniqueName: \"kubernetes.io/projected/74dd87a5-973e-4409-a688-1f1e5c0f840f-kube-api-access-ksl9g\") pod \"downloads-57bd96b888-gg6jn\" (UID: \"74dd87a5-973e-4409-a688-1f1e5c0f840f\") " pod="openshift-console/downloads-57bd96b888-gg6jn"
May 20 13:53:17.179744 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.179713    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vkftb"]
May 20 13:53:17.181645 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.181605    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.188224 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.188204    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 20 13:53:17.188370 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.188353    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 20 13:53:17.194962 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.194943    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 20 13:53:17.216399 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.216374    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 20 13:53:17.216527 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.216420    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tgnrd\""
May 20 13:53:17.230301 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.230268    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vkftb"]
May 20 13:53:17.275457 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.275430    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/923ac251-4722-4870-be54-9843ac346d9f-data-volume\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.275622 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.275463    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/923ac251-4722-4870-be54-9843ac346d9f-crio-socket\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.275622 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.275481    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/923ac251-4722-4870-be54-9843ac346d9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.275622 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.275502    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jg5\" (UniqueName: \"kubernetes.io/projected/923ac251-4722-4870-be54-9843ac346d9f-kube-api-access-d2jg5\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.275622 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.275595    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ksl9g\" (UniqueName: \"kubernetes.io/projected/74dd87a5-973e-4409-a688-1f1e5c0f840f-kube-api-access-ksl9g\") pod \"downloads-57bd96b888-gg6jn\" (UID: \"74dd87a5-973e-4409-a688-1f1e5c0f840f\") " pod="openshift-console/downloads-57bd96b888-gg6jn"
May 20 13:53:17.275765 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.275626    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/923ac251-4722-4870-be54-9843ac346d9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.297793 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.297767    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksl9g\" (UniqueName: \"kubernetes.io/projected/74dd87a5-973e-4409-a688-1f1e5c0f840f-kube-api-access-ksl9g\") pod \"downloads-57bd96b888-gg6jn\" (UID: \"74dd87a5-973e-4409-a688-1f1e5c0f840f\") " pod="openshift-console/downloads-57bd96b888-gg6jn"
May 20 13:53:17.376318 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.376280    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/923ac251-4722-4870-be54-9843ac346d9f-crio-socket\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.376318 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.376315    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/923ac251-4722-4870-be54-9843ac346d9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.376571 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.376332    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jg5\" (UniqueName: \"kubernetes.io/projected/923ac251-4722-4870-be54-9843ac346d9f-kube-api-access-d2jg5\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.376571 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.376386    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:53:17.376571 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.376406    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/923ac251-4722-4870-be54-9843ac346d9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.376571 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.376431    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/923ac251-4722-4870-be54-9843ac346d9f-crio-socket\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.376571 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.376460    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:53:17.376571 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.376494    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/923ac251-4722-4870-be54-9843ac346d9f-data-volume\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.376853 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.376809    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/923ac251-4722-4870-be54-9843ac346d9f-data-volume\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.376991 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.376963    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/923ac251-4722-4870-be54-9843ac346d9f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.377154 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.377130    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/daad92a8-291c-4a57-9bb3-40d58be05740-service-ca-bundle\") pod \"router-default-656dc888b4-j7s2n\" (UID: \"daad92a8-291c-4a57-9bb3-40d58be05740\") " pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:53:17.378642 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.378614    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-gg6jn"
May 20 13:53:17.378909 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.378887    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/923ac251-4722-4870-be54-9843ac346d9f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.378974 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.378950    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74f1e9a4-b0ef-48e5-8107-1b9a218414e1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-2xrcl\" (UID: \"74f1e9a4-b0ef-48e5-8107-1b9a218414e1\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:53:17.388961 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.388934    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jg5\" (UniqueName: \"kubernetes.io/projected/923ac251-4722-4870-be54-9843ac346d9f-kube-api-access-d2jg5\") pod \"insights-runtime-extractor-vkftb\" (UID: \"923ac251-4722-4870-be54-9843ac346d9f\") " pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.409167 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.409140    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"
May 20 13:53:17.490324 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.490141    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vkftb"
May 20 13:53:17.532921 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.532865    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-gg6jn"]
May 20 13:53:17.538291 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:17.538248    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74dd87a5_973e_4409_a688_1f1e5c0f840f.slice/crio-7e09961086fa2ba17418287869dbbbbbfcb1beea5bd575ebddacbb3d0b5d90af WatchSource:0}: Error finding container 7e09961086fa2ba17418287869dbbbbbfcb1beea5bd575ebddacbb3d0b5d90af: Status 404 returned error can't find the container with id 7e09961086fa2ba17418287869dbbbbbfcb1beea5bd575ebddacbb3d0b5d90af
May 20 13:53:17.562987 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.562931    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl"]
May 20 13:53:17.627444 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.627414    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:53:17.656990 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.656945    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vkftb"]
May 20 13:53:17.659441 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:17.659412    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923ac251_4722_4870_be54_9843ac346d9f.slice/crio-b5228b1b566ec067d70fde0ab6dc2251cd2ecf24d5f2adcddf49495b0f36b0c4 WatchSource:0}: Error finding container b5228b1b566ec067d70fde0ab6dc2251cd2ecf24d5f2adcddf49495b0f36b0c4: Status 404 returned error can't find the container with id b5228b1b566ec067d70fde0ab6dc2251cd2ecf24d5f2adcddf49495b0f36b0c4
May 20 13:53:17.768898 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.768868    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-656dc888b4-j7s2n"]
May 20 13:53:17.772481 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:17.772447    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaad92a8_291c_4a57_9bb3_40d58be05740.slice/crio-62e5340dfa003060c27de88e0ee5b56f1735b62dd6368308aa7f3e0c2c909078 WatchSource:0}: Error finding container 62e5340dfa003060c27de88e0ee5b56f1735b62dd6368308aa7f3e0c2c909078: Status 404 returned error can't find the container with id 62e5340dfa003060c27de88e0ee5b56f1735b62dd6368308aa7f3e0c2c909078
May 20 13:53:17.956627 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.956589    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-gg6jn" event={"ID":"74dd87a5-973e-4409-a688-1f1e5c0f840f","Type":"ContainerStarted","Data":"7e09961086fa2ba17418287869dbbbbbfcb1beea5bd575ebddacbb3d0b5d90af"}
May 20 13:53:17.957898 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.957866    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-656dc888b4-j7s2n" event={"ID":"daad92a8-291c-4a57-9bb3-40d58be05740","Type":"ContainerStarted","Data":"0088d8dc00db5dbf825479f0085ace0383caa6b64e263eba821f6a63146595ef"}
May 20 13:53:17.958040 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.957903    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-656dc888b4-j7s2n" event={"ID":"daad92a8-291c-4a57-9bb3-40d58be05740","Type":"ContainerStarted","Data":"62e5340dfa003060c27de88e0ee5b56f1735b62dd6368308aa7f3e0c2c909078"}
May 20 13:53:17.959495 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.959466    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl" event={"ID":"74f1e9a4-b0ef-48e5-8107-1b9a218414e1","Type":"ContainerStarted","Data":"a4d422e7fe866a49a46140e0bb0b998d806bd5814b325694db690b89623344a0"}
May 20 13:53:17.961010 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.960989    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkftb" event={"ID":"923ac251-4722-4870-be54-9843ac346d9f","Type":"ContainerStarted","Data":"0ea6301d600fb5ae0a2cf280d47167d5f05cdf03850198a1a7a34f125de38ae7"}
May 20 13:53:17.961100 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.961018    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkftb" event={"ID":"923ac251-4722-4870-be54-9843ac346d9f","Type":"ContainerStarted","Data":"b5228b1b566ec067d70fde0ab6dc2251cd2ecf24d5f2adcddf49495b0f36b0c4"}
May 20 13:53:17.984395 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:17.984298    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-656dc888b4-j7s2n" podStartSLOduration=32.984283705 podStartE2EDuration="32.984283705s" podCreationTimestamp="2026-05-20 13:52:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:53:17.983452083 +0000 UTC m=+176.109835236" watchObservedRunningTime="2026-05-20 13:53:17.984283705 +0000 UTC m=+176.110666857"
May 20 13:53:18.628288 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:18.628195    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:53:18.631388 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:18.631356    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:53:18.966119 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:18.966033    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkftb" event={"ID":"923ac251-4722-4870-be54-9843ac346d9f","Type":"ContainerStarted","Data":"e12894ba6c3cc1dca5033a7d038c85cdef2ec67b9778f3883605422889780384"}
May 20 13:53:18.966626 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:18.966267    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:53:18.967924 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:18.967904    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-656dc888b4-j7s2n"
May 20 13:53:20.973431 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:20.973394    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl" event={"ID":"74f1e9a4-b0ef-48e5-8107-1b9a218414e1","Type":"ContainerStarted","Data":"29a5f8fe834947df23a94bedef86e922937e1624c09f36f1892b2eb4c751b0f2"}
May 20 13:53:20.975404 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:20.975370    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkftb" event={"ID":"923ac251-4722-4870-be54-9843ac346d9f","Type":"ContainerStarted","Data":"16240ca098b5039464ec4b4ce21bda2865314aa127e1639b65c0a7c182d01890"}
May 20 13:53:20.990821 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:20.990752    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-2xrcl" podStartSLOduration=33.526744586 podStartE2EDuration="35.990689644s" podCreationTimestamp="2026-05-20 13:52:45 +0000 UTC" firstStartedPulling="2026-05-20 13:53:17.573718734 +0000 UTC m=+175.700101872" lastFinishedPulling="2026-05-20 13:53:20.037663783 +0000 UTC m=+178.164046930" observedRunningTime="2026-05-20 13:53:20.988932122 +0000 UTC m=+179.115315277" watchObservedRunningTime="2026-05-20 13:53:20.990689644 +0000 UTC m=+179.117072798"
May 20 13:53:21.010595 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:21.010538    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vkftb" podStartSLOduration=1.711787398 podStartE2EDuration="4.010521103s" podCreationTimestamp="2026-05-20 13:53:17 +0000 UTC" firstStartedPulling="2026-05-20 13:53:17.740323743 +0000 UTC m=+175.866706874" lastFinishedPulling="2026-05-20 13:53:20.039057449 +0000 UTC m=+178.165440579" observedRunningTime="2026-05-20 13:53:21.009620107 +0000 UTC m=+179.136003251" watchObservedRunningTime="2026-05-20 13:53:21.010521103 +0000 UTC m=+179.136904256"
May 20 13:53:24.635422 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.635382    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"]
May 20 13:53:24.640722 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.640678    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.645035 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.645002    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 20 13:53:24.645475 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.645447    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 20 13:53:24.645601 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.645490    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 20 13:53:24.646216 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.645556    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2wq8c\""
May 20 13:53:24.650864 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.650801    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"]
May 20 13:53:24.743808 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.743775    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9411521-d5d0-45c4-97e0-1c6cc722e20d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.743808 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.743814    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9411521-d5d0-45c4-97e0-1c6cc722e20d-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.744044 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.743920    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9411521-d5d0-45c4-97e0-1c6cc722e20d-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.744044 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.743979    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz6hs\" (UniqueName: \"kubernetes.io/projected/b9411521-d5d0-45c4-97e0-1c6cc722e20d-kube-api-access-vz6hs\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.845207 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.845162    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vz6hs\" (UniqueName: \"kubernetes.io/projected/b9411521-d5d0-45c4-97e0-1c6cc722e20d-kube-api-access-vz6hs\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.845424 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.845244    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9411521-d5d0-45c4-97e0-1c6cc722e20d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.845424 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.845277    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9411521-d5d0-45c4-97e0-1c6cc722e20d-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.845424 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.845355    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9411521-d5d0-45c4-97e0-1c6cc722e20d-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.846146 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.846122    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9411521-d5d0-45c4-97e0-1c6cc722e20d-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.848062 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.848020    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9411521-d5d0-45c4-97e0-1c6cc722e20d-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.848172 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.848082    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9411521-d5d0-45c4-97e0-1c6cc722e20d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.853885 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.853857    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz6hs\" (UniqueName: \"kubernetes.io/projected/b9411521-d5d0-45c4-97e0-1c6cc722e20d-kube-api-access-vz6hs\") pod \"prometheus-operator-7779bbdd78-tsjd8\" (UID: \"b9411521-d5d0-45c4-97e0-1c6cc722e20d\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:24.955768 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:24.955696    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"
May 20 13:53:25.093852 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:25.093817    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8"]
May 20 13:53:25.096760 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:25.096726    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9411521_d5d0_45c4_97e0_1c6cc722e20d.slice/crio-cab64250e45c5c77f8e9cfe0bed18ca71acf5314a0ea88333957283a2be5d351 WatchSource:0}: Error finding container cab64250e45c5c77f8e9cfe0bed18ca71acf5314a0ea88333957283a2be5d351: Status 404 returned error can't find the container with id cab64250e45c5c77f8e9cfe0bed18ca71acf5314a0ea88333957283a2be5d351
May 20 13:53:25.990746 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:25.990704    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8" event={"ID":"b9411521-d5d0-45c4-97e0-1c6cc722e20d","Type":"ContainerStarted","Data":"cab64250e45c5c77f8e9cfe0bed18ca71acf5314a0ea88333957283a2be5d351"}
May 20 13:53:26.995071 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:26.995034    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8" event={"ID":"b9411521-d5d0-45c4-97e0-1c6cc722e20d","Type":"ContainerStarted","Data":"99eee74389925599505220a218668cb3bf2546bfb06cbb2ab79dea334aa733fa"}
May 20 13:53:26.995556 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:26.995078    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8" event={"ID":"b9411521-d5d0-45c4-97e0-1c6cc722e20d","Type":"ContainerStarted","Data":"3dce43a9a15d7c2219b5d453f1eb7dd48ff1456626cf10209c4de3d17557e826"}
May 20 13:53:27.012978 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:27.012914    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-tsjd8" podStartSLOduration=1.728805666 podStartE2EDuration="3.012896458s" podCreationTimestamp="2026-05-20 13:53:24 +0000 UTC" firstStartedPulling="2026-05-20 13:53:25.098807761 +0000 UTC m=+183.225190892" lastFinishedPulling="2026-05-20 13:53:26.382898535 +0000 UTC m=+184.509281684" observedRunningTime="2026-05-20 13:53:27.012047914 +0000 UTC m=+185.138431097" watchObservedRunningTime="2026-05-20 13:53:27.012896458 +0000 UTC m=+185.139279604"
May 20 13:53:27.099240 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:27.099211    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:53:28.897252 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:28.897216    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"]
May 20 13:53:28.901822 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:28.901799    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:28.904542 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:28.904522    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 20 13:53:28.904663 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:28.904524    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 20 13:53:28.904663 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:28.904524    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jnsk5\""
May 20 13:53:28.909860 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:28.909841    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"]
May 20 13:53:29.032539 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.032497    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dmbqj"]
May 20 13:53:29.036482 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.036457    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"]
May 20 13:53:29.036647 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.036620    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.039040 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.039012    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 20 13:53:29.039159 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.039012    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 20 13:53:29.039159 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.039061    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vz65g\""
May 20 13:53:29.039349 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.039284    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 20 13:53:29.039796 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.039777    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.041577 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.041553    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"]
May 20 13:53:29.042301 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.042283    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 20 13:53:29.042443 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.042393    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 20 13:53:29.042628 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.042612    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 20 13:53:29.043483 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.043461    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gmgck\""
May 20 13:53:29.079373 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.079324    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.079786 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.079409    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4t7\" (UniqueName: \"kubernetes.io/projected/6a17c27f-6c21-4883-92a4-00888af609ad-kube-api-access-5s4t7\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.079786 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.079454    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.079786 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.079506    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a17c27f-6c21-4883-92a4-00888af609ad-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.180757 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.180671    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4t7\" (UniqueName: \"kubernetes.io/projected/6a17c27f-6c21-4883-92a4-00888af609ad-kube-api-access-5s4t7\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.180757 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.180717    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtkb\" (UniqueName: \"kubernetes.io/projected/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-api-access-djtkb\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.180757 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.180752    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-accelerators-collector-config\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.181028 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.180870    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-wtmp\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.181028 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.180917    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.181028 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.180988    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.181028 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181024    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a17c27f-6c21-4883-92a4-00888af609ad-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.181224 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:29.181037    2550 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 20 13:53:29.181224 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181053    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-metrics-client-ca\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.181224 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181083    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.181224 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:29.181105    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-tls podName:6a17c27f-6c21-4883-92a4-00888af609ad nodeName:}" failed. No retries permitted until 2026-05-20 13:53:29.681084675 +0000 UTC m=+187.807467808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-pmb5p" (UID: "6a17c27f-6c21-4883-92a4-00888af609ad") : secret "openshift-state-metrics-tls" not found
May 20 13:53:29.181224 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181182    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.181224 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181219    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.181579 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181247    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-textfile\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.181579 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181278    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-sys\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.181579 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181306    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.181579 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181390    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-root\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.181579 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181414    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wf96\" (UniqueName: \"kubernetes.io/projected/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-kube-api-access-2wf96\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.181579 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181465    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.181579 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181503    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.181579 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181529    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-tls\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.181936 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.181782    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a17c27f-6c21-4883-92a4-00888af609ad-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.184187 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.184167    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.190004 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.189964    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4t7\" (UniqueName: \"kubernetes.io/projected/6a17c27f-6c21-4883-92a4-00888af609ad-kube-api-access-5s4t7\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.282188 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282148    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-accelerators-collector-config\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282376 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282196    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-wtmp\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282376 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282259    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.282376 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282290    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-metrics-client-ca\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282376 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282316    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.282376 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282367    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282555 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282408    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.282555 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282431    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-wtmp\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282739 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282710    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-textfile\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282858 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282437    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-textfile\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282858 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282797    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-sys\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282858 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:29.282826    2550 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
May 20 13:53:29.282998 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282858    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-sys\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282998 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282873    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-accelerators-collector-config\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282998 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:29.282897    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-tls podName:05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:29.782876347 +0000 UTC m=+187.909259496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-dtrm7" (UID: "05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9") : secret "kube-state-metrics-tls" not found
May 20 13:53:29.282998 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282878    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-metrics-client-ca\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282998 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282930    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.282998 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282965    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-root\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.282998 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.282989    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2wf96\" (UniqueName: \"kubernetes.io/projected/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-kube-api-access-2wf96\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.283422 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.283022    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.283422 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.283046    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.283422 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.283066    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-root\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.283422 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.283052    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-tls\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.283422 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.283183    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-djtkb\" (UniqueName: \"kubernetes.io/projected/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-api-access-djtkb\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.283422 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.283223    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.283422 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:29.283269    2550 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 20 13:53:29.283422 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:29.283321    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-tls podName:7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79 nodeName:}" failed. No retries permitted until 2026-05-20 13:53:29.783306405 +0000 UTC m=+187.909689550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-tls") pod "node-exporter-dmbqj" (UID: "7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79") : secret "node-exporter-tls" not found
May 20 13:53:29.283422 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.283332    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.286151 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.286130    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.286317 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.286294    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.295033 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.295005    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wf96\" (UniqueName: \"kubernetes.io/projected/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-kube-api-access-2wf96\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.295173 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.295051    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtkb\" (UniqueName: \"kubernetes.io/projected/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-api-access-djtkb\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.686275 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.686237    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:29.686501 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:29.686438    2550 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 20 13:53:29.686573 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:29.686539    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-tls podName:6a17c27f-6c21-4883-92a4-00888af609ad nodeName:}" failed. No retries permitted until 2026-05-20 13:53:30.686513489 +0000 UTC m=+188.812896641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-pmb5p" (UID: "6a17c27f-6c21-4883-92a4-00888af609ad") : secret "openshift-state-metrics-tls" not found
May 20 13:53:29.787114 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.787076    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-tls\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.787300 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.787206    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.789898 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.789867    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79-node-exporter-tls\") pod \"node-exporter-dmbqj\" (UID: \"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79\") " pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.790036 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.789902    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-dtrm7\" (UID: \"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:29.949929 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.949851    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dmbqj"
May 20 13:53:29.956290 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:29.956073    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"
May 20 13:53:30.142284 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.142245    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:53:30.147848 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.147821    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.151429 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.151228    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 20 13:53:30.151429 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.151240    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 20 13:53:30.151429 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.151246    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 20 13:53:30.151429 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.151327    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 20 13:53:30.151964 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.151947    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 20 13:53:30.152064 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.151977    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 20 13:53:30.152621 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.152588    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 20 13:53:30.152751 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.152626    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7jxv6\""
May 20 13:53:30.152811 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.152792    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 20 13:53:30.174016 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.173984    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 20 13:53:30.174799 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.174773    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:53:30.191221 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191190    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-volume\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.191462 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191237    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.191462 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191267    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44fwz\" (UniqueName: \"kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-kube-api-access-44fwz\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.191462 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191307    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-out\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.191462 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191361    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.191462 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191436    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.191739 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191468    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.191739 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191516    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.191739 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191564    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-web-config\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.191739 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191599    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.191739 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191634    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.191739 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191726    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.192025 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.191776    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.292623 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.292574    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.292800 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.292686    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.292800 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:30.292709    2550 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 20 13:53:30.292800 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.292729    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-web-config\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.292800 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.292754    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.292800 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:30.292776    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-main-tls podName:a45b3b4a-bae0-42a1-826b-fbc5667c6e6e nodeName:}" failed. No retries permitted until 2026-05-20 13:53:30.792755368 +0000 UTC m=+188.919138523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e") : secret "alertmanager-main-tls" not found
May 20 13:53:30.293064 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.292808    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.293064 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.292837    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.293064 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.292878    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.293064 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.292950    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-volume\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.293064 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.292975    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.293064 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.293001    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-44fwz\" (UniqueName: \"kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-kube-api-access-44fwz\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.293064 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.293033    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-out\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.293064 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.293061    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.293761 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.293144    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.293761 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.293192    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.294241 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.294167    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.294745 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.294581    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.296502 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.296443    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.296609 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.296536    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-volume\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.297700 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.297660    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-web-config\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.299387 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.298320    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.299387 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.299320    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.299575 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.299487    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.299916 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.299874    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.300200 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.300181    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-out\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.303912 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.303888    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44fwz\" (UniqueName: \"kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-kube-api-access-44fwz\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.698251 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.698161    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:30.700953 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.700926    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a17c27f-6c21-4883-92a4-00888af609ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-pmb5p\" (UID: \"6a17c27f-6c21-4883-92a4-00888af609ad\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:30.712810 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.712784    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"
May 20 13:53:30.799368 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.799317    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:30.802177 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:30.802153    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:31.060554 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:31.060511    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:53:33.372736 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.372695    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7478978558-hqpxf"]
May 20 13:53:33.376993 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.376931    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.381093 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.380996    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-szj8h\""
May 20 13:53:33.381093 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.381000    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 20 13:53:33.381284 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.381096    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 20 13:53:33.381284 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.381239    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-aus1kbpnrl1qq\""
May 20 13:53:33.381284 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.381276    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 20 13:53:33.381452 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.381324    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 20 13:53:33.387820 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.387792    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7478978558-hqpxf"]
May 20 13:53:33.424637 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.424599    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa74be60-54d6-45f8-b61a-2109c576facb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.424812 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.424648    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9274h\" (UniqueName: \"kubernetes.io/projected/fa74be60-54d6-45f8-b61a-2109c576facb-kube-api-access-9274h\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.424812 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.424731    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fa74be60-54d6-45f8-b61a-2109c576facb-metrics-server-audit-profiles\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.424812 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.424784    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fa74be60-54d6-45f8-b61a-2109c576facb-audit-log\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.424944 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.424856    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fa74be60-54d6-45f8-b61a-2109c576facb-secret-metrics-server-client-certs\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.424944 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.424884    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fa74be60-54d6-45f8-b61a-2109c576facb-client-ca-bundle\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.425036 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.424967    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fa74be60-54d6-45f8-b61a-2109c576facb-secret-metrics-server-tls\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.526481 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.526442    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa74be60-54d6-45f8-b61a-2109c576facb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.526668 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.526489    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9274h\" (UniqueName: \"kubernetes.io/projected/fa74be60-54d6-45f8-b61a-2109c576facb-kube-api-access-9274h\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.526668 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.526530    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fa74be60-54d6-45f8-b61a-2109c576facb-metrics-server-audit-profiles\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.526668 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.526557    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fa74be60-54d6-45f8-b61a-2109c576facb-audit-log\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.526668 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.526599    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fa74be60-54d6-45f8-b61a-2109c576facb-secret-metrics-server-client-certs\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.526668 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.526624    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fa74be60-54d6-45f8-b61a-2109c576facb-client-ca-bundle\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.526905 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.526678    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fa74be60-54d6-45f8-b61a-2109c576facb-secret-metrics-server-tls\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.527113 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.527079    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fa74be60-54d6-45f8-b61a-2109c576facb-audit-log\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.527374 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.527325    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa74be60-54d6-45f8-b61a-2109c576facb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.527667 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.527643    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fa74be60-54d6-45f8-b61a-2109c576facb-metrics-server-audit-profiles\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.529700 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.529650    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fa74be60-54d6-45f8-b61a-2109c576facb-client-ca-bundle\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.529795 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.529728    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fa74be60-54d6-45f8-b61a-2109c576facb-secret-metrics-server-tls\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.529847 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.529833    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fa74be60-54d6-45f8-b61a-2109c576facb-secret-metrics-server-client-certs\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.535456 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.535428    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9274h\" (UniqueName: \"kubernetes.io/projected/fa74be60-54d6-45f8-b61a-2109c576facb-kube-api-access-9274h\") pod \"metrics-server-7478978558-hqpxf\" (UID: \"fa74be60-54d6-45f8-b61a-2109c576facb\") " pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.688567 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:33.688480    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:33.841036 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:33.840992    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3ff7ba_5b0b_4ad3_ad8c_e6c0da52ab79.slice/crio-b0a9c90621e3d7067924cc747df8994b1c9b0898e4b02fe399335d07d9828409 WatchSource:0}: Error finding container b0a9c90621e3d7067924cc747df8994b1c9b0898e4b02fe399335d07d9828409: Status 404 returned error can't find the container with id b0a9c90621e3d7067924cc747df8994b1c9b0898e4b02fe399335d07d9828409
May 20 13:53:34.008268 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:34.008225    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p"]
May 20 13:53:34.012581 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:34.012541    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a17c27f_6c21_4883_92a4_00888af609ad.slice/crio-7daa6c5b8cc744eff00d8488b0fa8e0f9231c8d340d264f120b1d9f2920744dc WatchSource:0}: Error finding container 7daa6c5b8cc744eff00d8488b0fa8e0f9231c8d340d264f120b1d9f2920744dc: Status 404 returned error can't find the container with id 7daa6c5b8cc744eff00d8488b0fa8e0f9231c8d340d264f120b1d9f2920744dc
May 20 13:53:34.020622 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:34.020587    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-gg6jn" event={"ID":"74dd87a5-973e-4409-a688-1f1e5c0f840f","Type":"ContainerStarted","Data":"db9eece86d3ac9008dc38146effa8b9a958c022ec6aa4b8dd74f1c5980fae124"}
May 20 13:53:34.020980 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:34.020939    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-gg6jn"
May 20 13:53:34.022003 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:34.021957    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p" event={"ID":"6a17c27f-6c21-4883-92a4-00888af609ad","Type":"ContainerStarted","Data":"7daa6c5b8cc744eff00d8488b0fa8e0f9231c8d340d264f120b1d9f2920744dc"}
May 20 13:53:34.022420 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:34.022400    2550 patch_prober.go:28] interesting pod/downloads-57bd96b888-gg6jn container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused" start-of-body=
May 20 13:53:34.022520 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:34.022450    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-57bd96b888-gg6jn" podUID="74dd87a5-973e-4409-a688-1f1e5c0f840f" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused"
May 20 13:53:34.023081 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:34.023059    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dmbqj" event={"ID":"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79","Type":"ContainerStarted","Data":"b0a9c90621e3d7067924cc747df8994b1c9b0898e4b02fe399335d07d9828409"}
May 20 13:53:34.038265 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:34.038155    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-gg6jn" podStartSLOduration=0.662911352 podStartE2EDuration="17.038134861s" podCreationTimestamp="2026-05-20 13:53:17 +0000 UTC" firstStartedPulling="2026-05-20 13:53:17.541248823 +0000 UTC m=+175.667631969" lastFinishedPulling="2026-05-20 13:53:33.916472345 +0000 UTC m=+192.042855478" observedRunningTime="2026-05-20 13:53:34.038106986 +0000 UTC m=+192.164490141" watchObservedRunningTime="2026-05-20 13:53:34.038134861 +0000 UTC m=+192.164518015"
May 20 13:53:34.241622 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:34.241571    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7"]
May 20 13:53:34.244548 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:34.244457    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c9bb3e_e318_4dcb_bcc3_832ff3f6d4a9.slice/crio-bbc4a1c99c12c81af95c21ed8a150b6bc7048c8cd20d9a55fdf529b4d0e3145b WatchSource:0}: Error finding container bbc4a1c99c12c81af95c21ed8a150b6bc7048c8cd20d9a55fdf529b4d0e3145b: Status 404 returned error can't find the container with id bbc4a1c99c12c81af95c21ed8a150b6bc7048c8cd20d9a55fdf529b4d0e3145b
May 20 13:53:34.244972 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:34.244883    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7478978558-hqpxf"]
May 20 13:53:34.248555 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:34.248528    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa74be60_54d6_45f8_b61a_2109c576facb.slice/crio-f7883f8b6305cc7f14589bee97b6c70a9a9f8003440b9400f9598fab348f5944 WatchSource:0}: Error finding container f7883f8b6305cc7f14589bee97b6c70a9a9f8003440b9400f9598fab348f5944: Status 404 returned error can't find the container with id f7883f8b6305cc7f14589bee97b6c70a9a9f8003440b9400f9598fab348f5944
May 20 13:53:34.249252 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:34.249200    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:53:34.251854 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:53:34.251823    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45b3b4a_bae0_42a1_826b_fbc5667c6e6e.slice/crio-9037a50065dbbc25aedab2c55c641753a2b9121408bb8764d020dd8ca570a455 WatchSource:0}: Error finding container 9037a50065dbbc25aedab2c55c641753a2b9121408bb8764d020dd8ca570a455: Status 404 returned error can't find the container with id 9037a50065dbbc25aedab2c55c641753a2b9121408bb8764d020dd8ca570a455
May 20 13:53:35.040602 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:35.040561    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p" event={"ID":"6a17c27f-6c21-4883-92a4-00888af609ad","Type":"ContainerStarted","Data":"50f5a319810d0faf7f0c7009df974c9af48cfd8bd61ef54e9b0b63ee24e20c1d"}
May 20 13:53:35.042493 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:35.042449    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p" event={"ID":"6a17c27f-6c21-4883-92a4-00888af609ad","Type":"ContainerStarted","Data":"35888dcdf01f3c596269de0bf773aa48af84ba2b86beae1b1862f509ed12d265"}
May 20 13:53:35.043923 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:35.043862    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7478978558-hqpxf" event={"ID":"fa74be60-54d6-45f8-b61a-2109c576facb","Type":"ContainerStarted","Data":"f7883f8b6305cc7f14589bee97b6c70a9a9f8003440b9400f9598fab348f5944"}
May 20 13:53:35.045929 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:35.045874    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerStarted","Data":"9037a50065dbbc25aedab2c55c641753a2b9121408bb8764d020dd8ca570a455"}
May 20 13:53:35.052264 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:35.052051    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7" event={"ID":"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9","Type":"ContainerStarted","Data":"bbc4a1c99c12c81af95c21ed8a150b6bc7048c8cd20d9a55fdf529b4d0e3145b"}
May 20 13:53:35.062851 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:35.062827    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-gg6jn"
May 20 13:53:39.065409 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:39.065367    2550 generic.go:358] "Generic (PLEG): container finished" podID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerID="10b90db4fd2856de4cea688acf43e559309f899070de90c92471678540dfa1ba" exitCode=0
May 20 13:53:39.066080 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:39.065443    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerDied","Data":"10b90db4fd2856de4cea688acf43e559309f899070de90c92471678540dfa1ba"}
May 20 13:53:39.068637 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:39.068537    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7" event={"ID":"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9","Type":"ContainerStarted","Data":"3c09e8e6348a43c0b7c86d0f9fa9f7cfc5146c363e6dbc663067a61dcab863dd"}
May 20 13:53:39.068637 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:39.068576    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7" event={"ID":"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9","Type":"ContainerStarted","Data":"4641a2a27188e585c7e3ffb55460359dbfd1f5c00635e739b69618dd18be09b3"}
May 20 13:53:39.068637 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:39.068590    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7" event={"ID":"05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9","Type":"ContainerStarted","Data":"a32a8999f1dde5185f14c41d8e8df6979f1af5355188101a60c56e54a67f99ab"}
May 20 13:53:39.070704 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:39.070674    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p" event={"ID":"6a17c27f-6c21-4883-92a4-00888af609ad","Type":"ContainerStarted","Data":"7ac1c12cdc775ab7e2f1dc5a9c67225b9c3a5508871f6bbd8a6da72193d137a5"}
May 20 13:53:39.072619 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:39.072595    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7478978558-hqpxf" event={"ID":"fa74be60-54d6-45f8-b61a-2109c576facb","Type":"ContainerStarted","Data":"71f341e90e01d5b36156d2b41ea5b25fb1b86f91d22026ed1e758542d320cc66"}
May 20 13:53:39.074580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:39.074517    2550 generic.go:358] "Generic (PLEG): container finished" podID="7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79" containerID="51dd201b8ab990e01fbeedc7297151addd6909abf85e8be9403ecba70c324b2c" exitCode=0
May 20 13:53:39.074580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:39.074555    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dmbqj" event={"ID":"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79","Type":"ContainerDied","Data":"51dd201b8ab990e01fbeedc7297151addd6909abf85e8be9403ecba70c324b2c"}
May 20 13:53:39.124759 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:39.124705    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-pmb5p" podStartSLOduration=6.8613384029999995 podStartE2EDuration="11.124683609s" podCreationTimestamp="2026-05-20 13:53:28 +0000 UTC" firstStartedPulling="2026-05-20 13:53:34.141751139 +0000 UTC m=+192.268134285" lastFinishedPulling="2026-05-20 13:53:38.405096355 +0000 UTC m=+196.531479491" observedRunningTime="2026-05-20 13:53:39.123820839 +0000 UTC m=+197.250203994" watchObservedRunningTime="2026-05-20 13:53:39.124683609 +0000 UTC m=+197.251066763"
May 20 13:53:39.163996 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:39.163864    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-dtrm7" podStartSLOduration=6.005886915 podStartE2EDuration="10.16384515s" podCreationTimestamp="2026-05-20 13:53:29 +0000 UTC" firstStartedPulling="2026-05-20 13:53:34.247223332 +0000 UTC m=+192.373606473" lastFinishedPulling="2026-05-20 13:53:38.40518156 +0000 UTC m=+196.531564708" observedRunningTime="2026-05-20 13:53:39.162490801 +0000 UTC m=+197.288873976" watchObservedRunningTime="2026-05-20 13:53:39.16384515 +0000 UTC m=+197.290228303"
May 20 13:53:40.082849 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:40.082776    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dmbqj" event={"ID":"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79","Type":"ContainerStarted","Data":"53c5f5ad59692ba96c83c544f74b4d9fa7066b3ea18ba643702e054642053cc0"}
May 20 13:53:40.082849 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:40.082824    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dmbqj" event={"ID":"7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79","Type":"ContainerStarted","Data":"5017d93714d50df50b7884158a1998cf1b8bd6dbcbd8205ca422a2fb4ee2055a"}
May 20 13:53:40.110032 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:40.109950    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dmbqj" podStartSLOduration=6.548516581 podStartE2EDuration="11.109899781s" podCreationTimestamp="2026-05-20 13:53:29 +0000 UTC" firstStartedPulling="2026-05-20 13:53:33.843490314 +0000 UTC m=+191.969873462" lastFinishedPulling="2026-05-20 13:53:38.404873526 +0000 UTC m=+196.531256662" observedRunningTime="2026-05-20 13:53:40.107770328 +0000 UTC m=+198.234153485" watchObservedRunningTime="2026-05-20 13:53:40.109899781 +0000 UTC m=+198.236282934"
May 20 13:53:40.111362 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:40.110733    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7478978558-hqpxf" podStartSLOduration=2.95096492 podStartE2EDuration="7.110724003s" podCreationTimestamp="2026-05-20 13:53:33 +0000 UTC" firstStartedPulling="2026-05-20 13:53:34.250403473 +0000 UTC m=+192.376786604" lastFinishedPulling="2026-05-20 13:53:38.410162555 +0000 UTC m=+196.536545687" observedRunningTime="2026-05-20 13:53:39.185581682 +0000 UTC m=+197.311964836" watchObservedRunningTime="2026-05-20 13:53:40.110724003 +0000 UTC m=+198.237107157"
May 20 13:53:42.092944 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.092911    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerStarted","Data":"d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538"}
May 20 13:53:42.093305 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.092950    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerStarted","Data":"75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc"}
May 20 13:53:42.093305 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.092965    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerStarted","Data":"430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213"}
May 20 13:53:42.093305 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.092978    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerStarted","Data":"dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6"}
May 20 13:53:42.093305 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.092991    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerStarted","Data":"dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9"}
May 20 13:53:42.113299 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.113249    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj" podUID="c03f37cb-a242-4358-966d-af487e7d58b9" containerName="registry" containerID="cri-o://373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2" gracePeriod=30
May 20 13:53:42.424237 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.424200    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:53:42.525568 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.525531    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-registry-certificates\") pod \"c03f37cb-a242-4358-966d-af487e7d58b9\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") "
May 20 13:53:42.525754 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.525588    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-trusted-ca\") pod \"c03f37cb-a242-4358-966d-af487e7d58b9\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") "
May 20 13:53:42.525754 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.525613    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls\") pod \"c03f37cb-a242-4358-966d-af487e7d58b9\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") "
May 20 13:53:42.525754 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.525640    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8pcj\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-kube-api-access-q8pcj\") pod \"c03f37cb-a242-4358-966d-af487e7d58b9\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") "
May 20 13:53:42.525754 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.525672    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-bound-sa-token\") pod \"c03f37cb-a242-4358-966d-af487e7d58b9\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") "
May 20 13:53:42.525754 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.525723    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c03f37cb-a242-4358-966d-af487e7d58b9-ca-trust-extracted\") pod \"c03f37cb-a242-4358-966d-af487e7d58b9\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") "
May 20 13:53:42.526017 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.525757    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-installation-pull-secrets\") pod \"c03f37cb-a242-4358-966d-af487e7d58b9\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") "
May 20 13:53:42.526017 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.525801    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-image-registry-private-configuration\") pod \"c03f37cb-a242-4358-966d-af487e7d58b9\" (UID: \"c03f37cb-a242-4358-966d-af487e7d58b9\") "
May 20 13:53:42.526323 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.526232    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c03f37cb-a242-4358-966d-af487e7d58b9" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:53:42.526323 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.526285    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c03f37cb-a242-4358-966d-af487e7d58b9" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:53:42.528631 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.528603    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c03f37cb-a242-4358-966d-af487e7d58b9" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:53:42.529224 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.529194    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c03f37cb-a242-4358-966d-af487e7d58b9" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:53:42.529322 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.529239    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c03f37cb-a242-4358-966d-af487e7d58b9" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:53:42.529408 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.529331    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-kube-api-access-q8pcj" (OuterVolumeSpecName: "kube-api-access-q8pcj") pod "c03f37cb-a242-4358-966d-af487e7d58b9" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9"). InnerVolumeSpecName "kube-api-access-q8pcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:53:42.529771 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.529742    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c03f37cb-a242-4358-966d-af487e7d58b9" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:53:42.536278 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.536234    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f37cb-a242-4358-966d-af487e7d58b9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c03f37cb-a242-4358-966d-af487e7d58b9" (UID: "c03f37cb-a242-4358-966d-af487e7d58b9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:53:42.627438 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.627328    2550 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c03f37cb-a242-4358-966d-af487e7d58b9-ca-trust-extracted\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:53:42.627438 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.627433    2550 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-installation-pull-secrets\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:53:42.627438 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.627445    2550 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c03f37cb-a242-4358-966d-af487e7d58b9-image-registry-private-configuration\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:53:42.627660 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.627456    2550 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-registry-certificates\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:53:42.627660 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.627466    2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c03f37cb-a242-4358-966d-af487e7d58b9-trusted-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:53:42.627660 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.627474    2550 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-registry-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:53:42.627660 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.627483    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8pcj\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-kube-api-access-q8pcj\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:53:42.627660 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:42.627491    2550 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c03f37cb-a242-4358-966d-af487e7d58b9-bound-sa-token\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:53:43.098742 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:43.098706    2550 generic.go:358] "Generic (PLEG): container finished" podID="c03f37cb-a242-4358-966d-af487e7d58b9" containerID="373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2" exitCode=0
May 20 13:53:43.099111 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:43.098771    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj"
May 20 13:53:43.099111 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:43.098789    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj" event={"ID":"c03f37cb-a242-4358-966d-af487e7d58b9","Type":"ContainerDied","Data":"373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2"}
May 20 13:53:43.099111 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:43.098824    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cb7476d84-xwppj" event={"ID":"c03f37cb-a242-4358-966d-af487e7d58b9","Type":"ContainerDied","Data":"cfc670682e4aa66c6637d4405e362f2dd4ed141ae714dc4ce1aa0e9866ddcbcc"}
May 20 13:53:43.099111 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:43.098839    2550 scope.go:117] "RemoveContainer" containerID="373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2"
May 20 13:53:43.107688 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:43.107645    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerStarted","Data":"a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21"}
May 20 13:53:43.115395 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:43.115254    2550 scope.go:117] "RemoveContainer" containerID="373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2"
May 20 13:53:43.115606 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:53:43.115583    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2\": container with ID starting with 373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2 not found: ID does not exist" containerID="373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2"
May 20 13:53:43.115700 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:43.115610    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2"} err="failed to get container status \"373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2\": rpc error: code = NotFound desc = could not find container \"373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2\": container with ID starting with 373dc463ce7332c665c0495804d43d901d09d22a78fac1ee666a47c93de9cab2 not found: ID does not exist"
May 20 13:53:43.135407 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:43.135351    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.42319973 podStartE2EDuration="13.135323573s" podCreationTimestamp="2026-05-20 13:53:30 +0000 UTC" firstStartedPulling="2026-05-20 13:53:34.254261654 +0000 UTC m=+192.380644799" lastFinishedPulling="2026-05-20 13:53:42.966385505 +0000 UTC m=+201.092768642" observedRunningTime="2026-05-20 13:53:43.132984889 +0000 UTC m=+201.259368047" watchObservedRunningTime="2026-05-20 13:53:43.135323573 +0000 UTC m=+201.261706725"
May 20 13:53:43.153790 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:43.153744    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cb7476d84-xwppj"]
May 20 13:53:43.156921 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:43.156896    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cb7476d84-xwppj"]
May 20 13:53:44.466580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:44.466545    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c03f37cb-a242-4358-966d-af487e7d58b9" path="/var/lib/kubelet/pods/c03f37cb-a242-4358-966d-af487e7d58b9/volumes"
May 20 13:53:53.689153 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:53.689112    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:53:53.689153 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:53:53.689159    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:54:13.693653 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:13.693621    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:54:13.697506 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:13.697481    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7478978558-hqpxf"
May 20 13:54:34.280645 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:34.280600    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:54:34.282877 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:34.282856    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be-metrics-certs\") pod \"network-metrics-daemon-rvjzj\" (UID: \"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be\") " pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:54:34.566051 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:34.565971    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vwf8m\""
May 20 13:54:34.573114 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:34.573093    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rvjzj"
May 20 13:54:34.700018 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:34.699878    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rvjzj"]
May 20 13:54:34.702642 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:54:34.702617    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda281ce9a_c6ff_4bbb_a2bc_ae0ed33679be.slice/crio-ac0518c12a089bdc460843cdc762310ee85b544be914f32ea7e0b400c04459a8 WatchSource:0}: Error finding container ac0518c12a089bdc460843cdc762310ee85b544be914f32ea7e0b400c04459a8: Status 404 returned error can't find the container with id ac0518c12a089bdc460843cdc762310ee85b544be914f32ea7e0b400c04459a8
May 20 13:54:35.266469 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:35.266437    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rvjzj" event={"ID":"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be","Type":"ContainerStarted","Data":"ac0518c12a089bdc460843cdc762310ee85b544be914f32ea7e0b400c04459a8"}
May 20 13:54:36.270697 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:36.270665    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rvjzj" event={"ID":"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be","Type":"ContainerStarted","Data":"2f221dc84ff078e084c8e3b9b6704dee288760f43b488ba42080e0be6ed28794"}
May 20 13:54:36.270697 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:36.270699    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rvjzj" event={"ID":"a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be","Type":"ContainerStarted","Data":"9d4b446796d863a95d58197caeacf4226bc2ab6bc58367cefea9842cbe2a644c"}
May 20 13:54:36.290056 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:36.290004    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rvjzj" podStartSLOduration=253.265403537 podStartE2EDuration="4m14.289987368s" podCreationTimestamp="2026-05-20 13:50:22 +0000 UTC" firstStartedPulling="2026-05-20 13:54:34.704329315 +0000 UTC m=+252.830712445" lastFinishedPulling="2026-05-20 13:54:35.728913133 +0000 UTC m=+253.855296276" observedRunningTime="2026-05-20 13:54:36.288605875 +0000 UTC m=+254.414989029" watchObservedRunningTime="2026-05-20 13:54:36.289987368 +0000 UTC m=+254.416370567"
May 20 13:54:49.310220 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:49.310142    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:49.310676 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:49.310649    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="alertmanager" containerID="cri-o://dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9" gracePeriod=120
May 20 13:54:49.310739 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:49.310704    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy-metric" containerID="cri-o://d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538" gracePeriod=120
May 20 13:54:49.310806 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:49.310773    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy" containerID="cri-o://75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc" gracePeriod=120
May 20 13:54:49.310877 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:49.310847    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy-web" containerID="cri-o://430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213" gracePeriod=120
May 20 13:54:49.310927 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:49.310917    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="config-reloader" containerID="cri-o://dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6" gracePeriod=120
May 20 13:54:49.311078 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:49.311015    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="prom-label-proxy" containerID="cri-o://a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21" gracePeriod=120
May 20 13:54:50.312492 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.312459    2550 generic.go:358] "Generic (PLEG): container finished" podID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerID="a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21" exitCode=0
May 20 13:54:50.312492 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.312483    2550 generic.go:358] "Generic (PLEG): container finished" podID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerID="d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538" exitCode=0
May 20 13:54:50.312492 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.312490    2550 generic.go:358] "Generic (PLEG): container finished" podID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerID="75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc" exitCode=0
May 20 13:54:50.312492 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.312496    2550 generic.go:358] "Generic (PLEG): container finished" podID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerID="dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6" exitCode=0
May 20 13:54:50.312492 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.312502    2550 generic.go:358] "Generic (PLEG): container finished" podID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerID="dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9" exitCode=0
May 20 13:54:50.312981 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.312540    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerDied","Data":"a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21"}
May 20 13:54:50.312981 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.312574    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerDied","Data":"d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538"}
May 20 13:54:50.312981 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.312584    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerDied","Data":"75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc"}
May 20 13:54:50.312981 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.312595    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerDied","Data":"dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6"}
May 20 13:54:50.312981 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.312603    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerDied","Data":"dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9"}
May 20 13:54:50.556751 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.556730    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:50.716452 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716329    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-metrics-client-ca\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.716452 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716443    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-main-db\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.716674 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716466    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.716674 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716494    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-trusted-ca-bundle\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.716674 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716524    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-tls-assets\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.716674 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716563    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-volume\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.716674 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716591    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-web\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.716674 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716618    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.716674 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716644    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-out\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.716674 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716677    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-web-config\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.717052 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716708    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-main-tls\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.717052 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716739    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44fwz\" (UniqueName: \"kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-kube-api-access-44fwz\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.717052 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716748    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:50.717052 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716776    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-cluster-tls-config\") pod \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\" (UID: \"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e\") "
May 20 13:54:50.717052 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716832    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:54:50.717052 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.716908    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 13:54:50.717362 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.717057    2550 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-metrics-client-ca\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.717362 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.717078    2550 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-main-db\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.717362 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.717095    2550 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.719736 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.719694    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:50.719874 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.719767    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-volume" (OuterVolumeSpecName: "config-volume") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:50.719961 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.719930    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:54:50.720045 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.720025    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-kube-api-access-44fwz" (OuterVolumeSpecName: "kube-api-access-44fwz") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "kube-api-access-44fwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 13:54:50.720140 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.720124    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:50.720501 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.720473    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:50.720590 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.720545    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-out" (OuterVolumeSpecName: "config-out") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 13:54:50.720590 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.720570    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:50.725398 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.725370    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:50.731473 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.731439    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-web-config" (OuterVolumeSpecName: "web-config") pod "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" (UID: "a45b3b4a-bae0-42a1-826b-fbc5667c6e6e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 13:54:50.817922 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.817887    2550 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-tls-assets\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.817922 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.817917    2550 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-volume\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.817922 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.817930    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.818147 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.817941    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.818147 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.817951    2550 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-config-out\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.818147 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.817960    2550 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-web-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.818147 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.817970    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-main-tls\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.818147 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.817979    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44fwz\" (UniqueName: \"kubernetes.io/projected/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-kube-api-access-44fwz\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.818147 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.817989    2550 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-cluster-tls-config\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:50.818147 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:50.817998    2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 13:54:51.318190 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.318153    2550 generic.go:358] "Generic (PLEG): container finished" podID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerID="430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213" exitCode=0
May 20 13:54:51.318671 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.318201    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerDied","Data":"430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213"}
May 20 13:54:51.318671 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.318224    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a45b3b4a-bae0-42a1-826b-fbc5667c6e6e","Type":"ContainerDied","Data":"9037a50065dbbc25aedab2c55c641753a2b9121408bb8764d020dd8ca570a455"}
May 20 13:54:51.318671 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.318240    2550 scope.go:117] "RemoveContainer" containerID="a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21"
May 20 13:54:51.318671 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.318322    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.325484 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.325462    2550 scope.go:117] "RemoveContainer" containerID="d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538"
May 20 13:54:51.332441 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.332418    2550 scope.go:117] "RemoveContainer" containerID="75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc"
May 20 13:54:51.338819 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.338801    2550 scope.go:117] "RemoveContainer" containerID="430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213"
May 20 13:54:51.343014 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.342992    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:51.345841 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.345823    2550 scope.go:117] "RemoveContainer" containerID="dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6"
May 20 13:54:51.350493 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.350465    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:51.354877 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.354857    2550 scope.go:117] "RemoveContainer" containerID="dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9"
May 20 13:54:51.361326 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.361311    2550 scope.go:117] "RemoveContainer" containerID="10b90db4fd2856de4cea688acf43e559309f899070de90c92471678540dfa1ba"
May 20 13:54:51.368033 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.368005    2550 scope.go:117] "RemoveContainer" containerID="a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21"
May 20 13:54:51.368291 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:54:51.368273    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21\": container with ID starting with a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21 not found: ID does not exist" containerID="a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21"
May 20 13:54:51.368361 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.368301    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21"} err="failed to get container status \"a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21\": rpc error: code = NotFound desc = could not find container \"a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21\": container with ID starting with a7314a6fcdef16eb3d4d54414ade87c0d6aec352972f7c8efb2174d5783f1e21 not found: ID does not exist"
May 20 13:54:51.368361 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.368321    2550 scope.go:117] "RemoveContainer" containerID="d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538"
May 20 13:54:51.368599 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:54:51.368583    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538\": container with ID starting with d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538 not found: ID does not exist" containerID="d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538"
May 20 13:54:51.368650 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.368606    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538"} err="failed to get container status \"d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538\": rpc error: code = NotFound desc = could not find container \"d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538\": container with ID starting with d93833dc785991b410889d31a4803157faeae1d75c6829b33935b0d848d18538 not found: ID does not exist"
May 20 13:54:51.368650 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.368622    2550 scope.go:117] "RemoveContainer" containerID="75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc"
May 20 13:54:51.368892 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:54:51.368875    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc\": container with ID starting with 75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc not found: ID does not exist" containerID="75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc"
May 20 13:54:51.368957 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.368900    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc"} err="failed to get container status \"75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc\": rpc error: code = NotFound desc = could not find container \"75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc\": container with ID starting with 75b92cc14459ee125fede39790945943b758915bb9d72c0f65108354a4be3cbc not found: ID does not exist"
May 20 13:54:51.368957 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.368922    2550 scope.go:117] "RemoveContainer" containerID="430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213"
May 20 13:54:51.369162 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:54:51.369145    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213\": container with ID starting with 430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213 not found: ID does not exist" containerID="430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213"
May 20 13:54:51.369204 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.369168    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213"} err="failed to get container status \"430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213\": rpc error: code = NotFound desc = could not find container \"430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213\": container with ID starting with 430aa03da78d49ab256c12b95755ffe7ab19bb890f52caf84d5cb297e7f12213 not found: ID does not exist"
May 20 13:54:51.369204 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.369184    2550 scope.go:117] "RemoveContainer" containerID="dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6"
May 20 13:54:51.369384 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:54:51.369367    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6\": container with ID starting with dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6 not found: ID does not exist" containerID="dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6"
May 20 13:54:51.369459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.369390    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6"} err="failed to get container status \"dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6\": rpc error: code = NotFound desc = could not find container \"dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6\": container with ID starting with dff09e7899df516211a7f457cbff160faf2b3c9b494c2ea30b8eab1cc67b1cc6 not found: ID does not exist"
May 20 13:54:51.369459 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.369410    2550 scope.go:117] "RemoveContainer" containerID="dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9"
May 20 13:54:51.369651 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:54:51.369635    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9\": container with ID starting with dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9 not found: ID does not exist" containerID="dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9"
May 20 13:54:51.369707 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.369656    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9"} err="failed to get container status \"dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9\": rpc error: code = NotFound desc = could not find container \"dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9\": container with ID starting with dbcdc3482a9d38fa31e56667026eaf20661d4e94e7e5c1f280f990254123bea9 not found: ID does not exist"
May 20 13:54:51.369707 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.369676    2550 scope.go:117] "RemoveContainer" containerID="10b90db4fd2856de4cea688acf43e559309f899070de90c92471678540dfa1ba"
May 20 13:54:51.369897 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:54:51.369883    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b90db4fd2856de4cea688acf43e559309f899070de90c92471678540dfa1ba\": container with ID starting with 10b90db4fd2856de4cea688acf43e559309f899070de90c92471678540dfa1ba not found: ID does not exist" containerID="10b90db4fd2856de4cea688acf43e559309f899070de90c92471678540dfa1ba"
May 20 13:54:51.369946 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.369903    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b90db4fd2856de4cea688acf43e559309f899070de90c92471678540dfa1ba"} err="failed to get container status \"10b90db4fd2856de4cea688acf43e559309f899070de90c92471678540dfa1ba\": rpc error: code = NotFound desc = could not find container \"10b90db4fd2856de4cea688acf43e559309f899070de90c92471678540dfa1ba\": container with ID starting with 10b90db4fd2856de4cea688acf43e559309f899070de90c92471678540dfa1ba not found: ID does not exist"
May 20 13:54:51.388700 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.388672    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:51.389103 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389088    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="prom-label-proxy"
May 20 13:54:51.389155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389105    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="prom-label-proxy"
May 20 13:54:51.389155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389115    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="config-reloader"
May 20 13:54:51.389155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389121    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="config-reloader"
May 20 13:54:51.389155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389128    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="alertmanager"
May 20 13:54:51.389155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389133    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="alertmanager"
May 20 13:54:51.389155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389149    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy-metric"
May 20 13:54:51.389155 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389154    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy-metric"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389162    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy-web"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389167    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy-web"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389175    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389180    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389186    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="init-config-reloader"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389191    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="init-config-reloader"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389198    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03f37cb-a242-4358-966d-af487e7d58b9" containerName="registry"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389203    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03f37cb-a242-4358-966d-af487e7d58b9" containerName="registry"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389250    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy-metric"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389258    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy-web"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389264    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="c03f37cb-a242-4358-966d-af487e7d58b9" containerName="registry"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389270    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="alertmanager"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389276    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="kube-rbac-proxy"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389283    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="config-reloader"
May 20 13:54:51.389580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.389289    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" containerName="prom-label-proxy"
May 20 13:54:51.392685 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.392669    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.395663 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.395505    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 20 13:54:51.395663 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.395509    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 20 13:54:51.395663 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.395544    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 20 13:54:51.395663 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.395564    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 20 13:54:51.395935 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.395695    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 20 13:54:51.395935 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.395819    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 20 13:54:51.395935 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.395896    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 20 13:54:51.396089 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.395942    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 20 13:54:51.396089 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.395955    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7jxv6\""
May 20 13:54:51.401599 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.401579    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 20 13:54:51.407810 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.407779    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:51.523239 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523195    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/788c6eee-255d-48dd-85aa-53a51ed02d73-config-out\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523239 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523237    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-config-volume\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523478 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523260    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/788c6eee-255d-48dd-85aa-53a51ed02d73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523478 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523307    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/788c6eee-255d-48dd-85aa-53a51ed02d73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523478 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523374    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/788c6eee-255d-48dd-85aa-53a51ed02d73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523478 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523401    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523478 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523447    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-web-config\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523631 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523499    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523631 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523555    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523631 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523579    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/788c6eee-255d-48dd-85aa-53a51ed02d73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523631 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523597    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523631 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523614    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.523769 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.523633    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7j48\" (UniqueName: \"kubernetes.io/projected/788c6eee-255d-48dd-85aa-53a51ed02d73-kube-api-access-h7j48\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.624913 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.624809    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-web-config\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.624913 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.624868    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.624913 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.624906    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.624913 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.624925    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/788c6eee-255d-48dd-85aa-53a51ed02d73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.625235 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.624940    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.625235 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.624958    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.625235 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.624975    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h7j48\" (UniqueName: \"kubernetes.io/projected/788c6eee-255d-48dd-85aa-53a51ed02d73-kube-api-access-h7j48\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.625235 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.624992    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/788c6eee-255d-48dd-85aa-53a51ed02d73-config-out\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.625235 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.625017    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-config-volume\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.627930 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.627896    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/788c6eee-255d-48dd-85aa-53a51ed02d73-config-out\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628047 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.627961    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/788c6eee-255d-48dd-85aa-53a51ed02d73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628047 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.627992    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/788c6eee-255d-48dd-85aa-53a51ed02d73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628047 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628034    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/788c6eee-255d-48dd-85aa-53a51ed02d73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628230 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628080    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628230 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628131    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-config-volume\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628230 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628180    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628230 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628208    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628456 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628254    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-web-config\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628456 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628318    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/788c6eee-255d-48dd-85aa-53a51ed02d73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628456 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628366    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628588 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628516    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/788c6eee-255d-48dd-85aa-53a51ed02d73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628778 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628754    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/788c6eee-255d-48dd-85aa-53a51ed02d73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628936 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628912    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/788c6eee-255d-48dd-85aa-53a51ed02d73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.628999 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.628962    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.629991 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.629975    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/788c6eee-255d-48dd-85aa-53a51ed02d73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.635320 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.635297    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7j48\" (UniqueName: \"kubernetes.io/projected/788c6eee-255d-48dd-85aa-53a51ed02d73-kube-api-access-h7j48\") pod \"alertmanager-main-0\" (UID: \"788c6eee-255d-48dd-85aa-53a51ed02d73\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.703435 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.703397    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 13:54:51.858176 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:51.858138    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 13:54:51.862095 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:54:51.862070    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788c6eee_255d_48dd_85aa_53a51ed02d73.slice/crio-f32649ecd3983b1e23a2f021ea3d87381b45a20ba91eeda6505c052d25391463 WatchSource:0}: Error finding container f32649ecd3983b1e23a2f021ea3d87381b45a20ba91eeda6505c052d25391463: Status 404 returned error can't find the container with id f32649ecd3983b1e23a2f021ea3d87381b45a20ba91eeda6505c052d25391463
May 20 13:54:52.322514 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:52.322479    2550 generic.go:358] "Generic (PLEG): container finished" podID="788c6eee-255d-48dd-85aa-53a51ed02d73" containerID="10bf11fe317d26893f0a2b8a3c4ccbf2aa93b1ae7cba40bc7fa637d6b9bd5ea2" exitCode=0
May 20 13:54:52.322900 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:52.322573    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"788c6eee-255d-48dd-85aa-53a51ed02d73","Type":"ContainerDied","Data":"10bf11fe317d26893f0a2b8a3c4ccbf2aa93b1ae7cba40bc7fa637d6b9bd5ea2"}
May 20 13:54:52.322900 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:52.322598    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"788c6eee-255d-48dd-85aa-53a51ed02d73","Type":"ContainerStarted","Data":"f32649ecd3983b1e23a2f021ea3d87381b45a20ba91eeda6505c052d25391463"}
May 20 13:54:52.467786 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:52.467352    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a45b3b4a-bae0-42a1-826b-fbc5667c6e6e" path="/var/lib/kubelet/pods/a45b3b4a-bae0-42a1-826b-fbc5667c6e6e/volumes"
May 20 13:54:53.329993 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.329958    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"788c6eee-255d-48dd-85aa-53a51ed02d73","Type":"ContainerStarted","Data":"dbec7d896e65e0f2c7faa84a97044db68eaf59db7e8fdc9a51868cb1a5fabcd3"}
May 20 13:54:53.329993 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.329993    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"788c6eee-255d-48dd-85aa-53a51ed02d73","Type":"ContainerStarted","Data":"c58b2f192818de57a6ad796bc180fd4c0089cc6bd47060af610403712084929f"}
May 20 13:54:53.330528 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.330002    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"788c6eee-255d-48dd-85aa-53a51ed02d73","Type":"ContainerStarted","Data":"69e265dc19670085ec4aee72ec345f5c2814d506adc33818532a32553855f764"}
May 20 13:54:53.330528 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.330011    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"788c6eee-255d-48dd-85aa-53a51ed02d73","Type":"ContainerStarted","Data":"9f4d3665c50ad3d3664dca7d3ec4b04006fe6310fe48fd58ced48005696156bc"}
May 20 13:54:53.330528 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.330019    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"788c6eee-255d-48dd-85aa-53a51ed02d73","Type":"ContainerStarted","Data":"ce285964b124c60efacdef52d84df1fadbe332bd5adbcd0242eea0fac2319585"}
May 20 13:54:53.330528 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.330027    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"788c6eee-255d-48dd-85aa-53a51ed02d73","Type":"ContainerStarted","Data":"bf36d424bf1cf65a4f925a89044def8c6ef67db1be11cb6d1535fe9a2eec6566"}
May 20 13:54:53.355660 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.355621    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5f9869f974-6pb2h"]
May 20 13:54:53.358376 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.358321    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.361580 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.361556    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 20 13:54:53.362017 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.361988    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 20 13:54:53.362353 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.362309    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 20 13:54:53.362678 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.362629    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ngqj4\""
May 20 13:54:53.363464 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.363446    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 20 13:54:53.363947 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.363716    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 20 13:54:53.366227 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.366034    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.366019762 podStartE2EDuration="2.366019762s" podCreationTimestamp="2026-05-20 13:54:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 13:54:53.364609429 +0000 UTC m=+271.490992583" watchObservedRunningTime="2026-05-20 13:54:53.366019762 +0000 UTC m=+271.492402916"
May 20 13:54:53.369296 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.369272    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 20 13:54:53.375314 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.375279    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f9869f974-6pb2h"]
May 20 13:54:53.544269 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.544227    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-secret-telemeter-client\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.544269 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.544269    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7g2\" (UniqueName: \"kubernetes.io/projected/13f70248-994e-4682-af6b-8e6cc7c863c0-kube-api-access-6z7g2\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.544538 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.544291    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f70248-994e-4682-af6b-8e6cc7c863c0-serving-certs-ca-bundle\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.544538 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.544364    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-federate-client-tls\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.544538 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.544387    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f70248-994e-4682-af6b-8e6cc7c863c0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.544538 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.544409    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-telemeter-client-tls\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.544538 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.544425    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.544538 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.544524    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13f70248-994e-4682-af6b-8e6cc7c863c0-metrics-client-ca\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.645952 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.645857    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-federate-client-tls\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.645952 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.645910    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f70248-994e-4682-af6b-8e6cc7c863c0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.646197 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.645985    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-telemeter-client-tls\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.646197 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.646022    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.646197 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.646065    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13f70248-994e-4682-af6b-8e6cc7c863c0-metrics-client-ca\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.646409 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.646200    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-secret-telemeter-client\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.646409 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.646241    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7g2\" (UniqueName: \"kubernetes.io/projected/13f70248-994e-4682-af6b-8e6cc7c863c0-kube-api-access-6z7g2\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.646409 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.646319    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f70248-994e-4682-af6b-8e6cc7c863c0-serving-certs-ca-bundle\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.647129 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.646891    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f70248-994e-4682-af6b-8e6cc7c863c0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.647129 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.647105    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13f70248-994e-4682-af6b-8e6cc7c863c0-metrics-client-ca\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.647308 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.647189    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13f70248-994e-4682-af6b-8e6cc7c863c0-serving-certs-ca-bundle\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.648760 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.648739    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-telemeter-client-tls\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.648866 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.648761    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-federate-client-tls\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.648866 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.648832    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.649229 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.649213    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/13f70248-994e-4682-af6b-8e6cc7c863c0-secret-telemeter-client\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.660613 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.660584    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7g2\" (UniqueName: \"kubernetes.io/projected/13f70248-994e-4682-af6b-8e6cc7c863c0-kube-api-access-6z7g2\") pod \"telemeter-client-5f9869f974-6pb2h\" (UID: \"13f70248-994e-4682-af6b-8e6cc7c863c0\") " pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.682614 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.682577    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h"
May 20 13:54:53.846787 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:53.846745    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f9869f974-6pb2h"]
May 20 13:54:53.849751 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:54:53.849724    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f70248_994e_4682_af6b_8e6cc7c863c0.slice/crio-466aad482b22292036b19c713d498e63e3bc43e3a53b17af0782b6e958811c7a WatchSource:0}: Error finding container 466aad482b22292036b19c713d498e63e3bc43e3a53b17af0782b6e958811c7a: Status 404 returned error can't find the container with id 466aad482b22292036b19c713d498e63e3bc43e3a53b17af0782b6e958811c7a
May 20 13:54:54.335086 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:54.335047    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h" event={"ID":"13f70248-994e-4682-af6b-8e6cc7c863c0","Type":"ContainerStarted","Data":"466aad482b22292036b19c713d498e63e3bc43e3a53b17af0782b6e958811c7a"}
May 20 13:54:56.342289 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:56.342250    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h" event={"ID":"13f70248-994e-4682-af6b-8e6cc7c863c0","Type":"ContainerStarted","Data":"fa4eb1ba31748c382113c3f554bbffb469d32f36ecaa1080fc79a918ad0e3254"}
May 20 13:54:56.342289 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:56.342291    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h" event={"ID":"13f70248-994e-4682-af6b-8e6cc7c863c0","Type":"ContainerStarted","Data":"642eded3acb2ca25ffacb354e9abc8f2015e78deba04e33b51ad6aaad660d2a0"}
May 20 13:54:56.342707 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:56.342300    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h" event={"ID":"13f70248-994e-4682-af6b-8e6cc7c863c0","Type":"ContainerStarted","Data":"e432ee9cc861b5f1e3a25b40d374b08108b58f0b51c3a2eb58309b2b1077cb9e"}
May 20 13:54:56.375395 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:54:56.375322    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5f9869f974-6pb2h" podStartSLOduration=1.654222817 podStartE2EDuration="3.375305947s" podCreationTimestamp="2026-05-20 13:54:53 +0000 UTC" firstStartedPulling="2026-05-20 13:54:53.85153348 +0000 UTC m=+271.977916614" lastFinishedPulling="2026-05-20 13:54:55.57261661 +0000 UTC m=+273.698999744" observedRunningTime="2026-05-20 13:54:56.373135109 +0000 UTC m=+274.499518271" watchObservedRunningTime="2026-05-20 13:54:56.375305947 +0000 UTC m=+274.501689099"
May 20 13:55:22.363200 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:55:22.363175    2550 kubelet.go:1628] "Image garbage collection succeeded"
May 20 13:57:54.767226 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.767128    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-2k8hl"]
May 20 13:57:54.770325 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.770304    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-2k8hl"
May 20 13:57:54.772928 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.772902    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 20 13:57:54.774021 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.774002    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
May 20 13:57:54.774125 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.774005    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 20 13:57:54.774125 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.774010    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 20 13:57:54.774125 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.774051    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-rxcft\""
May 20 13:57:54.778590 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.778568    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-2k8hl"]
May 20 13:57:54.843401 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.843355    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e6e235de-84ba-4427-8089-1f24f65d1362-certificates\") pod \"keda-admission-cf49989db-2k8hl\" (UID: \"e6e235de-84ba-4427-8089-1f24f65d1362\") " pod="openshift-keda/keda-admission-cf49989db-2k8hl"
May 20 13:57:54.843401 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.843407    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxmvw\" (UniqueName: \"kubernetes.io/projected/e6e235de-84ba-4427-8089-1f24f65d1362-kube-api-access-rxmvw\") pod \"keda-admission-cf49989db-2k8hl\" (UID: \"e6e235de-84ba-4427-8089-1f24f65d1362\") " pod="openshift-keda/keda-admission-cf49989db-2k8hl"
May 20 13:57:54.944440 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.944402    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e6e235de-84ba-4427-8089-1f24f65d1362-certificates\") pod \"keda-admission-cf49989db-2k8hl\" (UID: \"e6e235de-84ba-4427-8089-1f24f65d1362\") " pod="openshift-keda/keda-admission-cf49989db-2k8hl"
May 20 13:57:54.944440 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.944448    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rxmvw\" (UniqueName: \"kubernetes.io/projected/e6e235de-84ba-4427-8089-1f24f65d1362-kube-api-access-rxmvw\") pod \"keda-admission-cf49989db-2k8hl\" (UID: \"e6e235de-84ba-4427-8089-1f24f65d1362\") " pod="openshift-keda/keda-admission-cf49989db-2k8hl"
May 20 13:57:54.944662 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:57:54.944563    2550 projected.go:264] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
May 20 13:57:54.944662 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:57:54.944590    2550 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-cf49989db-2k8hl: secret "keda-admission-webhooks-certs" not found
May 20 13:57:54.944738 ip-10-0-129-117 kubenswrapper[2550]: E0520 13:57:54.944673    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6e235de-84ba-4427-8089-1f24f65d1362-certificates podName:e6e235de-84ba-4427-8089-1f24f65d1362 nodeName:}" failed. No retries permitted until 2026-05-20 13:57:55.444651535 +0000 UTC m=+453.571034676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e6e235de-84ba-4427-8089-1f24f65d1362-certificates") pod "keda-admission-cf49989db-2k8hl" (UID: "e6e235de-84ba-4427-8089-1f24f65d1362") : secret "keda-admission-webhooks-certs" not found
May 20 13:57:54.953908 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:54.953879    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxmvw\" (UniqueName: \"kubernetes.io/projected/e6e235de-84ba-4427-8089-1f24f65d1362-kube-api-access-rxmvw\") pod \"keda-admission-cf49989db-2k8hl\" (UID: \"e6e235de-84ba-4427-8089-1f24f65d1362\") " pod="openshift-keda/keda-admission-cf49989db-2k8hl"
May 20 13:57:55.447708 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:55.447671    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e6e235de-84ba-4427-8089-1f24f65d1362-certificates\") pod \"keda-admission-cf49989db-2k8hl\" (UID: \"e6e235de-84ba-4427-8089-1f24f65d1362\") " pod="openshift-keda/keda-admission-cf49989db-2k8hl"
May 20 13:57:55.450177 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:55.450146    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e6e235de-84ba-4427-8089-1f24f65d1362-certificates\") pod \"keda-admission-cf49989db-2k8hl\" (UID: \"e6e235de-84ba-4427-8089-1f24f65d1362\") " pod="openshift-keda/keda-admission-cf49989db-2k8hl"
May 20 13:57:55.680685 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:55.680656    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-2k8hl"
May 20 13:57:55.810171 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:55.810146    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-2k8hl"]
May 20 13:57:55.812806 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:57:55.812778    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e235de_84ba_4427_8089_1f24f65d1362.slice/crio-42b3f2ef6165a38dff46997cda2cd5bf6399a62fa05592ac72fea01a49cd282a WatchSource:0}: Error finding container 42b3f2ef6165a38dff46997cda2cd5bf6399a62fa05592ac72fea01a49cd282a: Status 404 returned error can't find the container with id 42b3f2ef6165a38dff46997cda2cd5bf6399a62fa05592ac72fea01a49cd282a
May 20 13:57:55.814246 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:55.814225    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 13:57:55.865497 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:55.865454    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-2k8hl" event={"ID":"e6e235de-84ba-4427-8089-1f24f65d1362","Type":"ContainerStarted","Data":"42b3f2ef6165a38dff46997cda2cd5bf6399a62fa05592ac72fea01a49cd282a"}
May 20 13:57:59.881164 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:59.881133    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-2k8hl" event={"ID":"e6e235de-84ba-4427-8089-1f24f65d1362","Type":"ContainerStarted","Data":"ce812055c033b833752741cecc38e539e9038fe854ed1dba848a4df5510ec920"}
May 20 13:57:59.881565 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:59.881234    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-2k8hl"
May 20 13:57:59.901083 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:57:59.901038    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-2k8hl" podStartSLOduration=2.436329643 podStartE2EDuration="5.901023346s" podCreationTimestamp="2026-05-20 13:57:54 +0000 UTC" firstStartedPulling="2026-05-20 13:57:55.814416412 +0000 UTC m=+453.940799543" lastFinishedPulling="2026-05-20 13:57:59.279110107 +0000 UTC m=+457.405493246" observedRunningTime="2026-05-20 13:57:59.899773203 +0000 UTC m=+458.026156358" watchObservedRunningTime="2026-05-20 13:57:59.901023346 +0000 UTC m=+458.027406499"
May 20 13:58:20.886794 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:58:20.886757    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-2k8hl"
May 20 13:59:05.828871 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:05.828839    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-2764s"]
May 20 13:59:05.832127 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:05.832111    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-2764s"
May 20 13:59:05.834764 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:05.834735    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
May 20 13:59:05.834898 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:05.834764    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
May 20 13:59:05.834898 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:05.834735    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sfw2h\""
May 20 13:59:05.842304 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:05.842277    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-2764s"]
May 20 13:59:05.919066 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:05.919020    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q7kb\" (UniqueName: \"kubernetes.io/projected/0f172e99-3057-4093-8f61-90a48716afa0-kube-api-access-6q7kb\") pod \"cert-manager-cainjector-68b757865b-2764s\" (UID: \"0f172e99-3057-4093-8f61-90a48716afa0\") " pod="cert-manager/cert-manager-cainjector-68b757865b-2764s"
May 20 13:59:05.919066 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:05.919074    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f172e99-3057-4093-8f61-90a48716afa0-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-2764s\" (UID: \"0f172e99-3057-4093-8f61-90a48716afa0\") " pod="cert-manager/cert-manager-cainjector-68b757865b-2764s"
May 20 13:59:06.020488 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:06.020454    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7kb\" (UniqueName: \"kubernetes.io/projected/0f172e99-3057-4093-8f61-90a48716afa0-kube-api-access-6q7kb\") pod \"cert-manager-cainjector-68b757865b-2764s\" (UID: \"0f172e99-3057-4093-8f61-90a48716afa0\") " pod="cert-manager/cert-manager-cainjector-68b757865b-2764s"
May 20 13:59:06.020644 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:06.020498    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f172e99-3057-4093-8f61-90a48716afa0-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-2764s\" (UID: \"0f172e99-3057-4093-8f61-90a48716afa0\") " pod="cert-manager/cert-manager-cainjector-68b757865b-2764s"
May 20 13:59:06.029092 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:06.029067    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f172e99-3057-4093-8f61-90a48716afa0-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-2764s\" (UID: \"0f172e99-3057-4093-8f61-90a48716afa0\") " pod="cert-manager/cert-manager-cainjector-68b757865b-2764s"
May 20 13:59:06.029241 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:06.029128    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q7kb\" (UniqueName: \"kubernetes.io/projected/0f172e99-3057-4093-8f61-90a48716afa0-kube-api-access-6q7kb\") pod \"cert-manager-cainjector-68b757865b-2764s\" (UID: \"0f172e99-3057-4093-8f61-90a48716afa0\") " pod="cert-manager/cert-manager-cainjector-68b757865b-2764s"
May 20 13:59:06.141691 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:06.141575    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-2764s"
May 20 13:59:06.269324 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:06.269298    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-2764s"]
May 20 13:59:06.271904 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:59:06.271872    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f172e99_3057_4093_8f61_90a48716afa0.slice/crio-fb1faf84e93f7601f039005a2449f28ca81c3e0d033eb863d6c0d735220ede28 WatchSource:0}: Error finding container fb1faf84e93f7601f039005a2449f28ca81c3e0d033eb863d6c0d735220ede28: Status 404 returned error can't find the container with id fb1faf84e93f7601f039005a2449f28ca81c3e0d033eb863d6c0d735220ede28
May 20 13:59:07.079825 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:07.079788    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-2764s" event={"ID":"0f172e99-3057-4093-8f61-90a48716afa0","Type":"ContainerStarted","Data":"fb1faf84e93f7601f039005a2449f28ca81c3e0d033eb863d6c0d735220ede28"}
May 20 13:59:10.090513 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:10.090473    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-2764s" event={"ID":"0f172e99-3057-4093-8f61-90a48716afa0","Type":"ContainerStarted","Data":"dffe67abc3797cd0b50b4830ac7576f17ccc41f0153a31570af1aae062f5bd2e"}
May 20 13:59:10.107141 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:10.107080    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-68b757865b-2764s" podStartSLOduration=1.808719682 podStartE2EDuration="5.107064947s" podCreationTimestamp="2026-05-20 13:59:05 +0000 UTC" firstStartedPulling="2026-05-20 13:59:06.273713707 +0000 UTC m=+524.400096837" lastFinishedPulling="2026-05-20 13:59:09.572058968 +0000 UTC m=+527.698442102" observedRunningTime="2026-05-20 13:59:10.104936895 +0000 UTC m=+528.231320039" watchObservedRunningTime="2026-05-20 13:59:10.107064947 +0000 UTC m=+528.233448100"
May 20 13:59:26.592578 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.592487    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj"]
May 20 13:59:26.596285 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.596263    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj"
May 20 13:59:26.599493 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.599406    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 20 13:59:26.599493 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.599433    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-79mhm\""
May 20 13:59:26.600529 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.600496    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 20 13:59:26.608166 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.608113    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj"]
May 20 13:59:26.695236 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.695199    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b90bc282-64a2-4236-be4a-885652b98896-tmp\") pod \"openshift-lws-operator-bfc7f696d-r8xpj\" (UID: \"b90bc282-64a2-4236-be4a-885652b98896\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj"
May 20 13:59:26.695435 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.695239    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv464\" (UniqueName: \"kubernetes.io/projected/b90bc282-64a2-4236-be4a-885652b98896-kube-api-access-bv464\") pod \"openshift-lws-operator-bfc7f696d-r8xpj\" (UID: \"b90bc282-64a2-4236-be4a-885652b98896\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj"
May 20 13:59:26.795766 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.795725    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b90bc282-64a2-4236-be4a-885652b98896-tmp\") pod \"openshift-lws-operator-bfc7f696d-r8xpj\" (UID: \"b90bc282-64a2-4236-be4a-885652b98896\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj"
May 20 13:59:26.795766 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.795771    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bv464\" (UniqueName: \"kubernetes.io/projected/b90bc282-64a2-4236-be4a-885652b98896-kube-api-access-bv464\") pod \"openshift-lws-operator-bfc7f696d-r8xpj\" (UID: \"b90bc282-64a2-4236-be4a-885652b98896\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj"
May 20 13:59:26.796109 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.796080    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b90bc282-64a2-4236-be4a-885652b98896-tmp\") pod \"openshift-lws-operator-bfc7f696d-r8xpj\" (UID: \"b90bc282-64a2-4236-be4a-885652b98896\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj"
May 20 13:59:26.805320 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.805288    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv464\" (UniqueName: \"kubernetes.io/projected/b90bc282-64a2-4236-be4a-885652b98896-kube-api-access-bv464\") pod \"openshift-lws-operator-bfc7f696d-r8xpj\" (UID: \"b90bc282-64a2-4236-be4a-885652b98896\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj"
May 20 13:59:26.909024 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:26.908929    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj"
May 20 13:59:27.043881 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:27.043828    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj"]
May 20 13:59:27.046169 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:59:27.046141    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb90bc282_64a2_4236_be4a_885652b98896.slice/crio-7a425df35ae5c8f2ed9fdb94faf410b65cdfc2bff486e5c37029b04dc1f7d751 WatchSource:0}: Error finding container 7a425df35ae5c8f2ed9fdb94faf410b65cdfc2bff486e5c37029b04dc1f7d751: Status 404 returned error can't find the container with id 7a425df35ae5c8f2ed9fdb94faf410b65cdfc2bff486e5c37029b04dc1f7d751
May 20 13:59:27.143169 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:27.143139    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj" event={"ID":"b90bc282-64a2-4236-be4a-885652b98896","Type":"ContainerStarted","Data":"7a425df35ae5c8f2ed9fdb94faf410b65cdfc2bff486e5c37029b04dc1f7d751"}
May 20 13:59:30.154375 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:30.154271    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj" event={"ID":"b90bc282-64a2-4236-be4a-885652b98896","Type":"ContainerStarted","Data":"95a64056d93759996cec7a17f36966c9177f387b23ac032b5027f5219dd00656"}
May 20 13:59:30.171322 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:30.171275    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-r8xpj" podStartSLOduration=1.337797514 podStartE2EDuration="4.171258943s" podCreationTimestamp="2026-05-20 13:59:26 +0000 UTC" firstStartedPulling="2026-05-20 13:59:27.04760879 +0000 UTC m=+545.173991921" lastFinishedPulling="2026-05-20 13:59:29.88107022 +0000 UTC m=+548.007453350" observedRunningTime="2026-05-20 13:59:30.169972924 +0000 UTC m=+548.296356078" watchObservedRunningTime="2026-05-20 13:59:30.171258943 +0000 UTC m=+548.297642097"
May 20 13:59:45.942827 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:45.942791    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"]
May 20 13:59:45.946101 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:45.946079    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:45.950057 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:45.950029    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nd5s7\""
May 20 13:59:45.950057 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:45.950048    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 20 13:59:45.950235 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:45.950052    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 20 13:59:45.950235 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:45.950070    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 20 13:59:45.955588 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:45.955566    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"]
May 20 13:59:46.073829 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.073763    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq9pq\" (UniqueName: \"kubernetes.io/projected/a5439963-59db-4d32-a3d9-aa94fa511462-kube-api-access-sq9pq\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.073829 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.073830    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5439963-59db-4d32-a3d9-aa94fa511462-cert\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.074097 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.073930    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a5439963-59db-4d32-a3d9-aa94fa511462-manager-config\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.074097 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.073987    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5439963-59db-4d32-a3d9-aa94fa511462-metrics-cert\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.174853 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.174810    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sq9pq\" (UniqueName: \"kubernetes.io/projected/a5439963-59db-4d32-a3d9-aa94fa511462-kube-api-access-sq9pq\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.174853 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.174856    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5439963-59db-4d32-a3d9-aa94fa511462-cert\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.175051 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.174904    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a5439963-59db-4d32-a3d9-aa94fa511462-manager-config\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.175051 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.174944    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5439963-59db-4d32-a3d9-aa94fa511462-metrics-cert\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.175594 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.175560    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a5439963-59db-4d32-a3d9-aa94fa511462-manager-config\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.177473 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.177454    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5439963-59db-4d32-a3d9-aa94fa511462-metrics-cert\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.177572 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.177479    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5439963-59db-4d32-a3d9-aa94fa511462-cert\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.185256 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.185213    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq9pq\" (UniqueName: \"kubernetes.io/projected/a5439963-59db-4d32-a3d9-aa94fa511462-kube-api-access-sq9pq\") pod \"lws-controller-manager-79d5949c76-nccdr\" (UID: \"a5439963-59db-4d32-a3d9-aa94fa511462\") " pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.257042 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.257007    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:46.384439 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:46.384413    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"]
May 20 13:59:46.387513 ip-10-0-129-117 kubenswrapper[2550]: W0520 13:59:46.387470    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5439963_59db_4d32_a3d9_aa94fa511462.slice/crio-11bc67b959b043b87002ea81eb6cb8159ea83f6afebbacef6799adb61cf9ca2b WatchSource:0}: Error finding container 11bc67b959b043b87002ea81eb6cb8159ea83f6afebbacef6799adb61cf9ca2b: Status 404 returned error can't find the container with id 11bc67b959b043b87002ea81eb6cb8159ea83f6afebbacef6799adb61cf9ca2b
May 20 13:59:47.208450 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:47.208413    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr" event={"ID":"a5439963-59db-4d32-a3d9-aa94fa511462","Type":"ContainerStarted","Data":"11bc67b959b043b87002ea81eb6cb8159ea83f6afebbacef6799adb61cf9ca2b"}
May 20 13:59:49.216689 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:49.216593    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr" event={"ID":"a5439963-59db-4d32-a3d9-aa94fa511462","Type":"ContainerStarted","Data":"90bd03804115d4f73a40022dee5ed9b206422e1c5d90259acf54ed71bb75abe5"}
May 20 13:59:49.216689 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:49.216652    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 13:59:49.236245 ip-10-0-129-117 kubenswrapper[2550]: I0520 13:59:49.236185    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr" podStartSLOduration=1.732552234 podStartE2EDuration="4.23616954s" podCreationTimestamp="2026-05-20 13:59:45 +0000 UTC" firstStartedPulling="2026-05-20 13:59:46.38937525 +0000 UTC m=+564.515758384" lastFinishedPulling="2026-05-20 13:59:48.892992559 +0000 UTC m=+567.019375690" observedRunningTime="2026-05-20 13:59:49.234403538 +0000 UTC m=+567.360786692" watchObservedRunningTime="2026-05-20 13:59:49.23616954 +0000 UTC m=+567.362552754"
May 20 14:00:00.221687 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:00.221656    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-79d5949c76-nccdr"
May 20 14:00:43.773138 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.773049    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"]
May 20 14:00:43.776490 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.776467    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:43.780380 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.780332    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
May 20 14:00:43.780380 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.780377    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 20 14:00:43.780909 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.780894    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
May 20 14:00:43.781518 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.781503    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 20 14:00:43.781599 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.781504    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-25lfs\""
May 20 14:00:43.785780 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.785760    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"]
May 20 14:00:43.871389 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.871324    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d838655f-8373-4ecf-9eb0-8d3a5572835b-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-r4lmp\" (UID: \"d838655f-8373-4ecf-9eb0-8d3a5572835b\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:43.871389 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.871388    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x9r7\" (UniqueName: \"kubernetes.io/projected/d838655f-8373-4ecf-9eb0-8d3a5572835b-kube-api-access-4x9r7\") pod \"kuadrant-console-plugin-6c886788f8-r4lmp\" (UID: \"d838655f-8373-4ecf-9eb0-8d3a5572835b\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:43.871609 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.871491    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d838655f-8373-4ecf-9eb0-8d3a5572835b-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-r4lmp\" (UID: \"d838655f-8373-4ecf-9eb0-8d3a5572835b\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:43.972702 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.972669    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d838655f-8373-4ecf-9eb0-8d3a5572835b-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-r4lmp\" (UID: \"d838655f-8373-4ecf-9eb0-8d3a5572835b\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:43.972902 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.972727    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d838655f-8373-4ecf-9eb0-8d3a5572835b-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-r4lmp\" (UID: \"d838655f-8373-4ecf-9eb0-8d3a5572835b\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:43.972902 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.972747    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4x9r7\" (UniqueName: \"kubernetes.io/projected/d838655f-8373-4ecf-9eb0-8d3a5572835b-kube-api-access-4x9r7\") pod \"kuadrant-console-plugin-6c886788f8-r4lmp\" (UID: \"d838655f-8373-4ecf-9eb0-8d3a5572835b\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:43.972902 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:00:43.972814    2550 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
May 20 14:00:43.972902 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:00:43.972902    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d838655f-8373-4ecf-9eb0-8d3a5572835b-plugin-serving-cert podName:d838655f-8373-4ecf-9eb0-8d3a5572835b nodeName:}" failed. No retries permitted until 2026-05-20 14:00:44.472880274 +0000 UTC m=+622.599263419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d838655f-8373-4ecf-9eb0-8d3a5572835b-plugin-serving-cert") pod "kuadrant-console-plugin-6c886788f8-r4lmp" (UID: "d838655f-8373-4ecf-9eb0-8d3a5572835b") : secret "plugin-serving-cert" not found
May 20 14:00:43.973284 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.973265    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d838655f-8373-4ecf-9eb0-8d3a5572835b-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-r4lmp\" (UID: \"d838655f-8373-4ecf-9eb0-8d3a5572835b\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:43.987171 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:43.987144    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x9r7\" (UniqueName: \"kubernetes.io/projected/d838655f-8373-4ecf-9eb0-8d3a5572835b-kube-api-access-4x9r7\") pod \"kuadrant-console-plugin-6c886788f8-r4lmp\" (UID: \"d838655f-8373-4ecf-9eb0-8d3a5572835b\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:44.477856 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:44.477817    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d838655f-8373-4ecf-9eb0-8d3a5572835b-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-r4lmp\" (UID: \"d838655f-8373-4ecf-9eb0-8d3a5572835b\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:44.478047 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:00:44.477963    2550 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
May 20 14:00:44.478047 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:00:44.478036    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d838655f-8373-4ecf-9eb0-8d3a5572835b-plugin-serving-cert podName:d838655f-8373-4ecf-9eb0-8d3a5572835b nodeName:}" failed. No retries permitted until 2026-05-20 14:00:45.478017751 +0000 UTC m=+623.604400896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d838655f-8373-4ecf-9eb0-8d3a5572835b-plugin-serving-cert") pod "kuadrant-console-plugin-6c886788f8-r4lmp" (UID: "d838655f-8373-4ecf-9eb0-8d3a5572835b") : secret "plugin-serving-cert" not found
May 20 14:00:45.486626 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:45.486585    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d838655f-8373-4ecf-9eb0-8d3a5572835b-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-r4lmp\" (UID: \"d838655f-8373-4ecf-9eb0-8d3a5572835b\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:45.489120 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:45.489096    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d838655f-8373-4ecf-9eb0-8d3a5572835b-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-r4lmp\" (UID: \"d838655f-8373-4ecf-9eb0-8d3a5572835b\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:45.586449 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:45.586406    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"
May 20 14:00:45.715302 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:45.715279    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp"]
May 20 14:00:45.717625 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:00:45.717593    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd838655f_8373_4ecf_9eb0_8d3a5572835b.slice/crio-b8698fbaf9fd98725814a7068b54b8fd6c989cfdc071f9ee7c03695cca62a7a8 WatchSource:0}: Error finding container b8698fbaf9fd98725814a7068b54b8fd6c989cfdc071f9ee7c03695cca62a7a8: Status 404 returned error can't find the container with id b8698fbaf9fd98725814a7068b54b8fd6c989cfdc071f9ee7c03695cca62a7a8
May 20 14:00:46.393523 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:46.393490    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp" event={"ID":"d838655f-8373-4ecf-9eb0-8d3a5572835b","Type":"ContainerStarted","Data":"b8698fbaf9fd98725814a7068b54b8fd6c989cfdc071f9ee7c03695cca62a7a8"}
May 20 14:00:51.413571 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:51.413533    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp" event={"ID":"d838655f-8373-4ecf-9eb0-8d3a5572835b","Type":"ContainerStarted","Data":"e96f7f128a730cc53ab13087a8fa98bba75cf19ee2b275bb8c074018b23f6658"}
May 20 14:00:51.430126 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:00:51.430068    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-r4lmp" podStartSLOduration=3.241372836 podStartE2EDuration="8.430049983s" podCreationTimestamp="2026-05-20 14:00:43 +0000 UTC" firstStartedPulling="2026-05-20 14:00:45.718929695 +0000 UTC m=+623.845312826" lastFinishedPulling="2026-05-20 14:00:50.907606625 +0000 UTC m=+629.033989973" observedRunningTime="2026-05-20 14:00:51.428941186 +0000 UTC m=+629.555324335" watchObservedRunningTime="2026-05-20 14:00:51.430049983 +0000 UTC m=+629.556433137"
May 20 14:01:25.760150 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:25.760109    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"]
May 20 14:01:25.763527 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:25.763511    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:25.766244 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:25.766222    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 20 14:01:25.773083 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:25.773057    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"]
May 20 14:01:25.859350 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:25.859307    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"]
May 20 14:01:25.938906 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:25.938871    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkcsw\" (UniqueName: \"kubernetes.io/projected/a9aed18e-24bc-4546-ae28-ed34618fbd8b-kube-api-access-dkcsw\") pod \"limitador-limitador-64c8f475fb-8f8xd\" (UID: \"a9aed18e-24bc-4546-ae28-ed34618fbd8b\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:25.939078 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:25.938928    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a9aed18e-24bc-4546-ae28-ed34618fbd8b-config-file\") pod \"limitador-limitador-64c8f475fb-8f8xd\" (UID: \"a9aed18e-24bc-4546-ae28-ed34618fbd8b\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:26.039817 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.039729    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dkcsw\" (UniqueName: \"kubernetes.io/projected/a9aed18e-24bc-4546-ae28-ed34618fbd8b-kube-api-access-dkcsw\") pod \"limitador-limitador-64c8f475fb-8f8xd\" (UID: \"a9aed18e-24bc-4546-ae28-ed34618fbd8b\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:26.039817 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.039789    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a9aed18e-24bc-4546-ae28-ed34618fbd8b-config-file\") pod \"limitador-limitador-64c8f475fb-8f8xd\" (UID: \"a9aed18e-24bc-4546-ae28-ed34618fbd8b\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:26.040400 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.040376    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a9aed18e-24bc-4546-ae28-ed34618fbd8b-config-file\") pod \"limitador-limitador-64c8f475fb-8f8xd\" (UID: \"a9aed18e-24bc-4546-ae28-ed34618fbd8b\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:26.047820 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.047792    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkcsw\" (UniqueName: \"kubernetes.io/projected/a9aed18e-24bc-4546-ae28-ed34618fbd8b-kube-api-access-dkcsw\") pod \"limitador-limitador-64c8f475fb-8f8xd\" (UID: \"a9aed18e-24bc-4546-ae28-ed34618fbd8b\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:26.074629 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.074599    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:26.203900 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.203875    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"]
May 20 14:01:26.206447 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:01:26.206405    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9aed18e_24bc_4546_ae28_ed34618fbd8b.slice/crio-8d8704747cb486a3ad291c8c287d4bcb20d7ace286219ac1f43eb40792eee4b7 WatchSource:0}: Error finding container 8d8704747cb486a3ad291c8c287d4bcb20d7ace286219ac1f43eb40792eee4b7: Status 404 returned error can't find the container with id 8d8704747cb486a3ad291c8c287d4bcb20d7ace286219ac1f43eb40792eee4b7
May 20 14:01:26.528642 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.528606    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd" event={"ID":"a9aed18e-24bc-4546-ae28-ed34618fbd8b","Type":"ContainerStarted","Data":"8d8704747cb486a3ad291c8c287d4bcb20d7ace286219ac1f43eb40792eee4b7"}
May 20 14:01:26.602595 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.602562    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-674b59b84c-kv47x"]
May 20 14:01:26.609284 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.609264    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-kv47x"
May 20 14:01:26.612231 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.612207    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qglng\""
May 20 14:01:26.615591 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.615566    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-kv47x"]
May 20 14:01:26.646020 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.645986    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqp7\" (UniqueName: \"kubernetes.io/projected/677a69ee-b832-4754-a72b-9b8a18de949d-kube-api-access-9gqp7\") pod \"authorino-674b59b84c-kv47x\" (UID: \"677a69ee-b832-4754-a72b-9b8a18de949d\") " pod="kuadrant-system/authorino-674b59b84c-kv47x"
May 20 14:01:26.746730 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.746695    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqp7\" (UniqueName: \"kubernetes.io/projected/677a69ee-b832-4754-a72b-9b8a18de949d-kube-api-access-9gqp7\") pod \"authorino-674b59b84c-kv47x\" (UID: \"677a69ee-b832-4754-a72b-9b8a18de949d\") " pod="kuadrant-system/authorino-674b59b84c-kv47x"
May 20 14:01:26.756043 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.756010    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqp7\" (UniqueName: \"kubernetes.io/projected/677a69ee-b832-4754-a72b-9b8a18de949d-kube-api-access-9gqp7\") pod \"authorino-674b59b84c-kv47x\" (UID: \"677a69ee-b832-4754-a72b-9b8a18de949d\") " pod="kuadrant-system/authorino-674b59b84c-kv47x"
May 20 14:01:26.920182 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:26.920097    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-kv47x"
May 20 14:01:27.073406 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:27.073378    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-kv47x"]
May 20 14:01:27.075411 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:01:27.075380    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod677a69ee_b832_4754_a72b_9b8a18de949d.slice/crio-d5a8d9f9e8fc85f8191b524ac8f0b4ab615e16ad511b6f29eeec191dec56b358 WatchSource:0}: Error finding container d5a8d9f9e8fc85f8191b524ac8f0b4ab615e16ad511b6f29eeec191dec56b358: Status 404 returned error can't find the container with id d5a8d9f9e8fc85f8191b524ac8f0b4ab615e16ad511b6f29eeec191dec56b358
May 20 14:01:27.533140 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:27.533096    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd" event={"ID":"a9aed18e-24bc-4546-ae28-ed34618fbd8b","Type":"ContainerStarted","Data":"00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376"}
May 20 14:01:27.533326 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:27.533162    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:27.534049 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:27.534031    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-kv47x" event={"ID":"677a69ee-b832-4754-a72b-9b8a18de949d","Type":"ContainerStarted","Data":"d5a8d9f9e8fc85f8191b524ac8f0b4ab615e16ad511b6f29eeec191dec56b358"}
May 20 14:01:27.549550 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:27.549481    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd" podStartSLOduration=1.3693159719999999 podStartE2EDuration="2.549466855s" podCreationTimestamp="2026-05-20 14:01:25 +0000 UTC" firstStartedPulling="2026-05-20 14:01:26.208303229 +0000 UTC m=+664.334686360" lastFinishedPulling="2026-05-20 14:01:27.388454106 +0000 UTC m=+665.514837243" observedRunningTime="2026-05-20 14:01:27.548826149 +0000 UTC m=+665.675209326" watchObservedRunningTime="2026-05-20 14:01:27.549466855 +0000 UTC m=+665.675850007"
May 20 14:01:29.726600 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:29.726560    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-kv47x"]
May 20 14:01:30.545296 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:30.545255    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-kv47x" event={"ID":"677a69ee-b832-4754-a72b-9b8a18de949d","Type":"ContainerStarted","Data":"1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114"}
May 20 14:01:30.559528 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:30.559472    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-674b59b84c-kv47x" podStartSLOduration=2.077214854 podStartE2EDuration="4.559456931s" podCreationTimestamp="2026-05-20 14:01:26 +0000 UTC" firstStartedPulling="2026-05-20 14:01:27.077001118 +0000 UTC m=+665.203384249" lastFinishedPulling="2026-05-20 14:01:29.559243192 +0000 UTC m=+667.685626326" observedRunningTime="2026-05-20 14:01:30.558851536 +0000 UTC m=+668.685234689" watchObservedRunningTime="2026-05-20 14:01:30.559456931 +0000 UTC m=+668.685840083"
May 20 14:01:31.551751 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:31.551709    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-674b59b84c-kv47x" podUID="677a69ee-b832-4754-a72b-9b8a18de949d" containerName="authorino" containerID="cri-o://1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114" gracePeriod=30
May 20 14:01:31.803143 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:31.803075    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-kv47x"
May 20 14:01:31.891809 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:31.891768    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gqp7\" (UniqueName: \"kubernetes.io/projected/677a69ee-b832-4754-a72b-9b8a18de949d-kube-api-access-9gqp7\") pod \"677a69ee-b832-4754-a72b-9b8a18de949d\" (UID: \"677a69ee-b832-4754-a72b-9b8a18de949d\") "
May 20 14:01:31.894017 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:31.893989    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677a69ee-b832-4754-a72b-9b8a18de949d-kube-api-access-9gqp7" (OuterVolumeSpecName: "kube-api-access-9gqp7") pod "677a69ee-b832-4754-a72b-9b8a18de949d" (UID: "677a69ee-b832-4754-a72b-9b8a18de949d"). InnerVolumeSpecName "kube-api-access-9gqp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:01:31.993326 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:31.993287    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gqp7\" (UniqueName: \"kubernetes.io/projected/677a69ee-b832-4754-a72b-9b8a18de949d-kube-api-access-9gqp7\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:01:32.555614 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:32.555580    2550 generic.go:358] "Generic (PLEG): container finished" podID="677a69ee-b832-4754-a72b-9b8a18de949d" containerID="1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114" exitCode=0
May 20 14:01:32.556017 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:32.555634    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-kv47x"
May 20 14:01:32.556017 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:32.555646    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-kv47x" event={"ID":"677a69ee-b832-4754-a72b-9b8a18de949d","Type":"ContainerDied","Data":"1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114"}
May 20 14:01:32.556017 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:32.555678    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-kv47x" event={"ID":"677a69ee-b832-4754-a72b-9b8a18de949d","Type":"ContainerDied","Data":"d5a8d9f9e8fc85f8191b524ac8f0b4ab615e16ad511b6f29eeec191dec56b358"}
May 20 14:01:32.556017 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:32.555695    2550 scope.go:117] "RemoveContainer" containerID="1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114"
May 20 14:01:32.564000 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:32.563980    2550 scope.go:117] "RemoveContainer" containerID="1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114"
May 20 14:01:32.564298 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:01:32.564260    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114\": container with ID starting with 1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114 not found: ID does not exist" containerID="1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114"
May 20 14:01:32.564298 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:32.564291    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114"} err="failed to get container status \"1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114\": rpc error: code = NotFound desc = could not find container \"1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114\": container with ID starting with 1073a50b0c3c02c28361a01d23edd55473cdab303eec639d0ad4539334151114 not found: ID does not exist"
May 20 14:01:32.577529 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:32.577499    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-kv47x"]
May 20 14:01:32.578916 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:32.578896    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-674b59b84c-kv47x"]
May 20 14:01:34.466967 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:34.466934    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="677a69ee-b832-4754-a72b-9b8a18de949d" path="/var/lib/kubelet/pods/677a69ee-b832-4754-a72b-9b8a18de949d/volumes"
May 20 14:01:38.538471 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:38.538444    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:42.623393 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:42.623352    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"]
May 20 14:01:42.623827 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:42.623630    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd" podUID="a9aed18e-24bc-4546-ae28-ed34618fbd8b" containerName="limitador" containerID="cri-o://00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376" gracePeriod=30
May 20 14:01:43.172530 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.172489    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:43.289700 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.289668    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a9aed18e-24bc-4546-ae28-ed34618fbd8b-config-file\") pod \"a9aed18e-24bc-4546-ae28-ed34618fbd8b\" (UID: \"a9aed18e-24bc-4546-ae28-ed34618fbd8b\") "
May 20 14:01:43.289855 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.289758    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkcsw\" (UniqueName: \"kubernetes.io/projected/a9aed18e-24bc-4546-ae28-ed34618fbd8b-kube-api-access-dkcsw\") pod \"a9aed18e-24bc-4546-ae28-ed34618fbd8b\" (UID: \"a9aed18e-24bc-4546-ae28-ed34618fbd8b\") "
May 20 14:01:43.290048 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.290025    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9aed18e-24bc-4546-ae28-ed34618fbd8b-config-file" (OuterVolumeSpecName: "config-file") pod "a9aed18e-24bc-4546-ae28-ed34618fbd8b" (UID: "a9aed18e-24bc-4546-ae28-ed34618fbd8b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 14:01:43.291853 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.291830    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9aed18e-24bc-4546-ae28-ed34618fbd8b-kube-api-access-dkcsw" (OuterVolumeSpecName: "kube-api-access-dkcsw") pod "a9aed18e-24bc-4546-ae28-ed34618fbd8b" (UID: "a9aed18e-24bc-4546-ae28-ed34618fbd8b"). InnerVolumeSpecName "kube-api-access-dkcsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:01:43.390924 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.390886    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkcsw\" (UniqueName: \"kubernetes.io/projected/a9aed18e-24bc-4546-ae28-ed34618fbd8b-kube-api-access-dkcsw\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:01:43.390924 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.390917    2550 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a9aed18e-24bc-4546-ae28-ed34618fbd8b-config-file\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:01:43.591357 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.591250    2550 generic.go:358] "Generic (PLEG): container finished" podID="a9aed18e-24bc-4546-ae28-ed34618fbd8b" containerID="00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376" exitCode=0
May 20 14:01:43.591357 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.591310    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"
May 20 14:01:43.591357 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.591320    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd" event={"ID":"a9aed18e-24bc-4546-ae28-ed34618fbd8b","Type":"ContainerDied","Data":"00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376"}
May 20 14:01:43.591627 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.591373    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-8f8xd" event={"ID":"a9aed18e-24bc-4546-ae28-ed34618fbd8b","Type":"ContainerDied","Data":"8d8704747cb486a3ad291c8c287d4bcb20d7ace286219ac1f43eb40792eee4b7"}
May 20 14:01:43.591627 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.591389    2550 scope.go:117] "RemoveContainer" containerID="00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376"
May 20 14:01:43.599594 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.599577    2550 scope.go:117] "RemoveContainer" containerID="00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376"
May 20 14:01:43.599846 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:01:43.599824    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376\": container with ID starting with 00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376 not found: ID does not exist" containerID="00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376"
May 20 14:01:43.599896 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.599855    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376"} err="failed to get container status \"00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376\": rpc error: code = NotFound desc = could not find container \"00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376\": container with ID starting with 00a25eb0c1d1c3265384a80f139d589197071c47f79e5a235fd5abcf781d7376 not found: ID does not exist"
May 20 14:01:43.613574 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.613543    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"]
May 20 14:01:43.619147 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:43.619122    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-8f8xd"]
May 20 14:01:44.467182 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:01:44.467149    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9aed18e-24bc-4546-ae28-ed34618fbd8b" path="/var/lib/kubelet/pods/a9aed18e-24bc-4546-ae28-ed34618fbd8b/volumes"
May 20 14:02:01.513608 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.513560    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"]
May 20 14:02:01.514205 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.514025    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677a69ee-b832-4754-a72b-9b8a18de949d" containerName="authorino"
May 20 14:02:01.514205 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.514044    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="677a69ee-b832-4754-a72b-9b8a18de949d" containerName="authorino"
May 20 14:02:01.514205 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.514089    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9aed18e-24bc-4546-ae28-ed34618fbd8b" containerName="limitador"
May 20 14:02:01.514205 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.514099    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9aed18e-24bc-4546-ae28-ed34618fbd8b" containerName="limitador"
May 20 14:02:01.514205 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.514180    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="a9aed18e-24bc-4546-ae28-ed34618fbd8b" containerName="limitador"
May 20 14:02:01.514205 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.514192    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="677a69ee-b832-4754-a72b-9b8a18de949d" containerName="authorino"
May 20 14:02:01.517571 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.517537    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.521700 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.521541    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
May 20 14:02:01.521700 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.521576    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
May 20 14:02:01.521895 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.521841    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
May 20 14:02:01.521942 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.521841    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zcsgc\""
May 20 14:02:01.522136 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.522107    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
May 20 14:02:01.534998 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.534971    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"]
May 20 14:02:01.646849 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.646799    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-token\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.646849 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.646841    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.647063 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.646867    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.647063 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.646965    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f825add0-7ef6-4b98-9794-ee03badebc8f-local-certs\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.647063 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.647003    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.647063 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.647037    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd89q\" (UniqueName: \"kubernetes.io/projected/f825add0-7ef6-4b98-9794-ee03badebc8f-kube-api-access-qd89q\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.647189 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.647090    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f825add0-7ef6-4b98-9794-ee03badebc8f-cacerts\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.748026 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.747985    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qd89q\" (UniqueName: \"kubernetes.io/projected/f825add0-7ef6-4b98-9794-ee03badebc8f-kube-api-access-qd89q\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.748210 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.748043    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f825add0-7ef6-4b98-9794-ee03badebc8f-cacerts\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.748210 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.748104    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-token\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.748210 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.748123    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.748210 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.748155    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.748210 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.748207    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f825add0-7ef6-4b98-9794-ee03badebc8f-local-certs\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.748486 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.748247    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.748897 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.748848    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.750526 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.750491    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f825add0-7ef6-4b98-9794-ee03badebc8f-local-certs\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.750677 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.750660    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.750931 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.750910    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.750999 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.750947    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f825add0-7ef6-4b98-9794-ee03badebc8f-cacerts\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.756668 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.756641    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f825add0-7ef6-4b98-9794-ee03badebc8f-istio-token\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.756791 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.756772    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd89q\" (UniqueName: \"kubernetes.io/projected/f825add0-7ef6-4b98-9794-ee03badebc8f-kube-api-access-qd89q\") pod \"istiod-openshift-gateway-798958bb55-mncdp\" (UID: \"f825add0-7ef6-4b98-9794-ee03badebc8f\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.829351 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.829311    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:01.979372 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:01.979329    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"]
May 20 14:02:01.981314 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:02:01.981280    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf825add0_7ef6_4b98_9794_ee03badebc8f.slice/crio-3a5fdf228fee7e849cf8f24dc3f03bd8611a51eb5dbd398d2e6ddf2ff4809302 WatchSource:0}: Error finding container 3a5fdf228fee7e849cf8f24dc3f03bd8611a51eb5dbd398d2e6ddf2ff4809302: Status 404 returned error can't find the container with id 3a5fdf228fee7e849cf8f24dc3f03bd8611a51eb5dbd398d2e6ddf2ff4809302
May 20 14:02:02.656418 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:02.656376    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp" event={"ID":"f825add0-7ef6-4b98-9794-ee03badebc8f","Type":"ContainerStarted","Data":"3a5fdf228fee7e849cf8f24dc3f03bd8611a51eb5dbd398d2e6ddf2ff4809302"}
May 20 14:02:05.507227 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:05.507184    2550 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:02:05.507549 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:05.507258    2550 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
May 20 14:02:05.667765 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:05.667722    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp" event={"ID":"f825add0-7ef6-4b98-9794-ee03badebc8f","Type":"ContainerStarted","Data":"c2b3909075733219c85c6c1d0287f17b4a595fabc12662c7bdd0b8ade4f4e2c5"}
May 20 14:02:05.667961 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:05.667852    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:02:05.745930 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:05.745857    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp" podStartSLOduration=1.222022528 podStartE2EDuration="4.7458365s" podCreationTimestamp="2026-05-20 14:02:01 +0000 UTC" firstStartedPulling="2026-05-20 14:02:01.983118463 +0000 UTC m=+700.109501595" lastFinishedPulling="2026-05-20 14:02:05.506932433 +0000 UTC m=+703.633315567" observedRunningTime="2026-05-20 14:02:05.743680778 +0000 UTC m=+703.870063948" watchObservedRunningTime="2026-05-20 14:02:05.7458365 +0000 UTC m=+703.872219652"
May 20 14:02:06.672687 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:02:06.672655    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-mncdp"
May 20 14:05:34.259144 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.259104    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"]
May 20 14:05:34.261654 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.261633    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.265040 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.265019    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:05:34.265142 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.265063    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:05:34.266963 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.266944    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:05:34.267066 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.267028    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:05:34.267134 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.267080    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
May 20 14:05:34.273229 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.273204    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"]
May 20 14:05:34.389416 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.389375    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.389416 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.389423    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.389645 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.389453    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.389645 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.389522    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657e83b6-730c-481f-9509-ffa3e5cd5e79-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.389645 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.389564    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.389645 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.389582    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7dzk\" (UniqueName: \"kubernetes.io/projected/657e83b6-730c-481f-9509-ffa3e5cd5e79-kube-api-access-v7dzk\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.389787 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.389675    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.490414 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.490369    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657e83b6-730c-481f-9509-ffa3e5cd5e79-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.490613 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.490439    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.490613 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.490459    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v7dzk\" (UniqueName: \"kubernetes.io/projected/657e83b6-730c-481f-9509-ffa3e5cd5e79-kube-api-access-v7dzk\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.490613 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.490506    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.490613 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.490529    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.490613 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.490562    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.490613 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.490581    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.491013 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.490983    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.491143 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.491033    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.491143 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.491117    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.491245 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.491155    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.492670 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.492652    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.492943 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.492926    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657e83b6-730c-481f-9509-ffa3e5cd5e79-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.498737 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.498699    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7dzk\" (UniqueName: \"kubernetes.io/projected/657e83b6-730c-481f-9509-ffa3e5cd5e79-kube-api-access-v7dzk\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.573451 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.573358    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:34.706715 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.706686    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"]
May 20 14:05:34.708900 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:05:34.708869    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657e83b6_730c_481f_9509_ffa3e5cd5e79.slice/crio-8b089f23b71832fe88760092852f9217931ee7a5f6fe1d1cd727478cc5239fa1 WatchSource:0}: Error finding container 8b089f23b71832fe88760092852f9217931ee7a5f6fe1d1cd727478cc5239fa1: Status 404 returned error can't find the container with id 8b089f23b71832fe88760092852f9217931ee7a5f6fe1d1cd727478cc5239fa1
May 20 14:05:34.710684 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:34.710664    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:05:35.373693 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:35.373661    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd" event={"ID":"657e83b6-730c-481f-9509-ffa3e5cd5e79","Type":"ContainerStarted","Data":"8b089f23b71832fe88760092852f9217931ee7a5f6fe1d1cd727478cc5239fa1"}
May 20 14:05:38.387016 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:38.386969    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd" event={"ID":"657e83b6-730c-481f-9509-ffa3e5cd5e79","Type":"ContainerStarted","Data":"05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a"}
May 20 14:05:42.402938 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:42.402900    2550 generic.go:358] "Generic (PLEG): container finished" podID="657e83b6-730c-481f-9509-ffa3e5cd5e79" containerID="05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a" exitCode=0
May 20 14:05:42.403389 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:42.402953    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd" event={"ID":"657e83b6-730c-481f-9509-ffa3e5cd5e79","Type":"ContainerDied","Data":"05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a"}
May 20 14:05:44.411408 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:44.411370    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd" event={"ID":"657e83b6-730c-481f-9509-ffa3e5cd5e79","Type":"ContainerStarted","Data":"c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf"}
May 20 14:05:44.431666 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:44.431617    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd" podStartSLOduration=1.434064805 podStartE2EDuration="10.431601592s" podCreationTimestamp="2026-05-20 14:05:34 +0000 UTC" firstStartedPulling="2026-05-20 14:05:34.710830698 +0000 UTC m=+912.837213829" lastFinishedPulling="2026-05-20 14:05:43.708367481 +0000 UTC m=+921.834750616" observedRunningTime="2026-05-20 14:05:44.43008442 +0000 UTC m=+922.556467583" watchObservedRunningTime="2026-05-20 14:05:44.431601592 +0000 UTC m=+922.557984745"
May 20 14:05:44.573772 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:44.573733    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:44.573772 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:44.573773    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:44.586292 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:44.586260    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:45.426196 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:45.426168    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:48.843853 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:48.843812    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"]
May 20 14:05:48.844266 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:48.844071    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd" podUID="657e83b6-730c-481f-9509-ffa3e5cd5e79" containerName="main" containerID="cri-o://c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf" gracePeriod=30
May 20 14:05:49.114772 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.114747    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:49.223394 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223329    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-home\") pod \"657e83b6-730c-481f-9509-ffa3e5cd5e79\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") "
May 20 14:05:49.223586 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223415    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-dshm\") pod \"657e83b6-730c-481f-9509-ffa3e5cd5e79\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") "
May 20 14:05:49.223586 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223474    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-model-cache\") pod \"657e83b6-730c-481f-9509-ffa3e5cd5e79\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") "
May 20 14:05:49.223586 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223517    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7dzk\" (UniqueName: \"kubernetes.io/projected/657e83b6-730c-481f-9509-ffa3e5cd5e79-kube-api-access-v7dzk\") pod \"657e83b6-730c-481f-9509-ffa3e5cd5e79\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") "
May 20 14:05:49.223586 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223561    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-kserve-provision-location\") pod \"657e83b6-730c-481f-9509-ffa3e5cd5e79\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") "
May 20 14:05:49.223586 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223586    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657e83b6-730c-481f-9509-ffa3e5cd5e79-tls-certs\") pod \"657e83b6-730c-481f-9509-ffa3e5cd5e79\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") "
May 20 14:05:49.223844 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223595    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-home" (OuterVolumeSpecName: "home") pod "657e83b6-730c-481f-9509-ffa3e5cd5e79" (UID: "657e83b6-730c-481f-9509-ffa3e5cd5e79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:05:49.223844 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223647    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-tmp-dir\") pod \"657e83b6-730c-481f-9509-ffa3e5cd5e79\" (UID: \"657e83b6-730c-481f-9509-ffa3e5cd5e79\") "
May 20 14:05:49.223844 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223760    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-model-cache" (OuterVolumeSpecName: "model-cache") pod "657e83b6-730c-481f-9509-ffa3e5cd5e79" (UID: "657e83b6-730c-481f-9509-ffa3e5cd5e79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:05:49.223962 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223905    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:05:49.223962 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223923    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:05:49.224069 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.223980    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "657e83b6-730c-481f-9509-ffa3e5cd5e79" (UID: "657e83b6-730c-481f-9509-ffa3e5cd5e79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:05:49.225836 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.225809    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657e83b6-730c-481f-9509-ffa3e5cd5e79-kube-api-access-v7dzk" (OuterVolumeSpecName: "kube-api-access-v7dzk") pod "657e83b6-730c-481f-9509-ffa3e5cd5e79" (UID: "657e83b6-730c-481f-9509-ffa3e5cd5e79"). InnerVolumeSpecName "kube-api-access-v7dzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:05:49.225943 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.225905    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/657e83b6-730c-481f-9509-ffa3e5cd5e79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "657e83b6-730c-481f-9509-ffa3e5cd5e79" (UID: "657e83b6-730c-481f-9509-ffa3e5cd5e79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:05:49.225998 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.225976    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-dshm" (OuterVolumeSpecName: "dshm") pod "657e83b6-730c-481f-9509-ffa3e5cd5e79" (UID: "657e83b6-730c-481f-9509-ffa3e5cd5e79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:05:49.290860 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.290818    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "657e83b6-730c-481f-9509-ffa3e5cd5e79" (UID: "657e83b6-730c-481f-9509-ffa3e5cd5e79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:05:49.325389 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.325353    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:05:49.325389 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.325388    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:05:49.325602 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.325402    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7dzk\" (UniqueName: \"kubernetes.io/projected/657e83b6-730c-481f-9509-ffa3e5cd5e79-kube-api-access-v7dzk\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:05:49.325602 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.325415    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657e83b6-730c-481f-9509-ffa3e5cd5e79-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:05:49.325602 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.325426    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657e83b6-730c-481f-9509-ffa3e5cd5e79-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:05:49.431668 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.431592    2550 generic.go:358] "Generic (PLEG): container finished" podID="657e83b6-730c-481f-9509-ffa3e5cd5e79" containerID="c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf" exitCode=0
May 20 14:05:49.431668 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.431651    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd" event={"ID":"657e83b6-730c-481f-9509-ffa3e5cd5e79","Type":"ContainerDied","Data":"c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf"}
May 20 14:05:49.431840 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.431696    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd" event={"ID":"657e83b6-730c-481f-9509-ffa3e5cd5e79","Type":"ContainerDied","Data":"8b089f23b71832fe88760092852f9217931ee7a5f6fe1d1cd727478cc5239fa1"}
May 20 14:05:49.431840 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.431711    2550 scope.go:117] "RemoveContainer" containerID="c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf"
May 20 14:05:49.431840 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.431669    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"
May 20 14:05:49.440777 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.440757    2550 scope.go:117] "RemoveContainer" containerID="05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a"
May 20 14:05:49.453832 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.453803    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"]
May 20 14:05:49.457788 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.457763    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-746bf7d5f-nzdcd"]
May 20 14:05:49.855485 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.855292    2550 scope.go:117] "RemoveContainer" containerID="c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf"
May 20 14:05:49.855758 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:05:49.855694    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf\": container with ID starting with c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf not found: ID does not exist" containerID="c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf"
May 20 14:05:49.855758 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.855720    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf"} err="failed to get container status \"c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf\": rpc error: code = NotFound desc = could not find container \"c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf\": container with ID starting with c7f6c5c41f63c741dec883d51c09b429901f8abd531faca7aac7180bce0ba9cf not found: ID does not exist"
May 20 14:05:49.855758 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.855740    2550 scope.go:117] "RemoveContainer" containerID="05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a"
May 20 14:05:49.856032 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:05:49.856008    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a\": container with ID starting with 05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a not found: ID does not exist" containerID="05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a"
May 20 14:05:49.856079 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:49.856042    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a"} err="failed to get container status \"05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a\": rpc error: code = NotFound desc = could not find container \"05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a\": container with ID starting with 05c2e03820cdbb37defbccf68572d8b1878e4b33ced83faf4776856311be818a not found: ID does not exist"
May 20 14:05:50.467286 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:05:50.467251    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="657e83b6-730c-481f-9509-ffa3e5cd5e79" path="/var/lib/kubelet/pods/657e83b6-730c-481f-9509-ffa3e5cd5e79/volumes"
May 20 14:06:01.465615 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.465577    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"]
May 20 14:06:01.466054 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.465965    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657e83b6-730c-481f-9509-ffa3e5cd5e79" containerName="storage-initializer"
May 20 14:06:01.466054 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.465981    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="657e83b6-730c-481f-9509-ffa3e5cd5e79" containerName="storage-initializer"
May 20 14:06:01.466054 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.466003    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657e83b6-730c-481f-9509-ffa3e5cd5e79" containerName="main"
May 20 14:06:01.466054 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.466009    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="657e83b6-730c-481f-9509-ffa3e5cd5e79" containerName="main"
May 20 14:06:01.466185 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.466060    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="657e83b6-730c-481f-9509-ffa3e5cd5e79" containerName="main"
May 20 14:06:01.479820 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.479785    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"]
May 20 14:06:01.479982 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.479893    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.482932 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.482904    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:06:01.483057 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.482943    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:06:01.483057 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.482904    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:06:01.484052 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.484029    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\""
May 20 14:06:01.484139 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.484035    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:06:01.530911 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.530871    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.530911 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.530916    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.531114 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.530939    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.531114 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.531028    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.531215 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.531186    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5chcg\" (UniqueName: \"kubernetes.io/projected/be4b2143-e746-45b5-ac54-c24f945cc204-kube-api-access-5chcg\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.531259 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.531232    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.531318 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.531267    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be4b2143-e746-45b5-ac54-c24f945cc204-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.631956 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.631920    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5chcg\" (UniqueName: \"kubernetes.io/projected/be4b2143-e746-45b5-ac54-c24f945cc204-kube-api-access-5chcg\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.632135 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.631962    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.632135 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.632087    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be4b2143-e746-45b5-ac54-c24f945cc204-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.632244 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.632144    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.632244 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.632190    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.632331 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.632234    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.632331 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.632303    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.632456 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.632386    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.632559 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.632542    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.632673 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.632651    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.632719 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.632704    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.634526 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.634483    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.634754 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.634732    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be4b2143-e746-45b5-ac54-c24f945cc204-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.640421 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.640393    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5chcg\" (UniqueName: \"kubernetes.io/projected/be4b2143-e746-45b5-ac54-c24f945cc204-kube-api-access-5chcg\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.792920 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.792888    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:06:01.930444 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:01.930410    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"]
May 20 14:06:01.933103 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:06:01.933075    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4b2143_e746_45b5_ac54_c24f945cc204.slice/crio-28e1452ca207c12711db24e81e8d39a8b79a6592e03bea4c028ca3b7cd5bd34f WatchSource:0}: Error finding container 28e1452ca207c12711db24e81e8d39a8b79a6592e03bea4c028ca3b7cd5bd34f: Status 404 returned error can't find the container with id 28e1452ca207c12711db24e81e8d39a8b79a6592e03bea4c028ca3b7cd5bd34f
May 20 14:06:02.475568 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:02.475530    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw" event={"ID":"be4b2143-e746-45b5-ac54-c24f945cc204","Type":"ContainerStarted","Data":"0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556"}
May 20 14:06:02.475568 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:06:02.475572    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw" event={"ID":"be4b2143-e746-45b5-ac54-c24f945cc204","Type":"ContainerStarted","Data":"28e1452ca207c12711db24e81e8d39a8b79a6592e03bea4c028ca3b7cd5bd34f"}
May 20 14:07:42.818104 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:42.818067    2550 generic.go:358] "Generic (PLEG): container finished" podID="be4b2143-e746-45b5-ac54-c24f945cc204" containerID="0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556" exitCode=0
May 20 14:07:42.818529 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:42.818142    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw" event={"ID":"be4b2143-e746-45b5-ac54-c24f945cc204","Type":"ContainerDied","Data":"0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556"}
May 20 14:07:43.823046 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:43.823012    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw" event={"ID":"be4b2143-e746-45b5-ac54-c24f945cc204","Type":"ContainerStarted","Data":"e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e"}
May 20 14:07:43.842692 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:43.842643    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw" podStartSLOduration=102.842627943 podStartE2EDuration="1m42.842627943s" podCreationTimestamp="2026-05-20 14:06:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:07:43.840693018 +0000 UTC m=+1041.967076170" watchObservedRunningTime="2026-05-20 14:07:43.842627943 +0000 UTC m=+1041.969011095"
May 20 14:07:51.793691 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:51.793651    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:07:51.793691 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:51.793691    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:07:51.806215 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:51.806184    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:07:51.861657 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:51.861626    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:07:52.863750 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:52.863716    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"]
May 20 14:07:53.019004 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:07:53.018966    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs: secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 20 14:07:53.019202 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:07:53.019069    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4b2143-e746-45b5-ac54-c24f945cc204-tls-certs podName:be4b2143-e746-45b5-ac54-c24f945cc204 nodeName:}" failed. No retries permitted until 2026-05-20 14:07:53.519045642 +0000 UTC m=+1051.645428775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/be4b2143-e746-45b5-ac54-c24f945cc204-tls-certs") pod "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw" (UID: "be4b2143-e746-45b5-ac54-c24f945cc204") : secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 20 14:07:53.523709 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:07:53.523678    2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs: secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 20 14:07:53.523890 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:07:53.523759    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4b2143-e746-45b5-ac54-c24f945cc204-tls-certs podName:be4b2143-e746-45b5-ac54-c24f945cc204 nodeName:}" failed. No retries permitted until 2026-05-20 14:07:54.523744238 +0000 UTC m=+1052.650127376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/be4b2143-e746-45b5-ac54-c24f945cc204-tls-certs") pod "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw" (UID: "be4b2143-e746-45b5-ac54-c24f945cc204") : secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 20 14:07:53.857914 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:53.857812    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw" podUID="be4b2143-e746-45b5-ac54-c24f945cc204" containerName="main" containerID="cri-o://e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e" gracePeriod=30
May 20 14:07:54.113968 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.113898    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:07:54.229596 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.229548    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-dshm\") pod \"be4b2143-e746-45b5-ac54-c24f945cc204\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") "
May 20 14:07:54.229596 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.229601    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5chcg\" (UniqueName: \"kubernetes.io/projected/be4b2143-e746-45b5-ac54-c24f945cc204-kube-api-access-5chcg\") pod \"be4b2143-e746-45b5-ac54-c24f945cc204\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") "
May 20 14:07:54.229865 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.229639    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-tmp-dir\") pod \"be4b2143-e746-45b5-ac54-c24f945cc204\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") "
May 20 14:07:54.229865 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.229742    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-home\") pod \"be4b2143-e746-45b5-ac54-c24f945cc204\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") "
May 20 14:07:54.229865 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.229775    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-model-cache\") pod \"be4b2143-e746-45b5-ac54-c24f945cc204\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") "
May 20 14:07:54.229865 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.229836    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be4b2143-e746-45b5-ac54-c24f945cc204-tls-certs\") pod \"be4b2143-e746-45b5-ac54-c24f945cc204\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") "
May 20 14:07:54.230065 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.229871    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-kserve-provision-location\") pod \"be4b2143-e746-45b5-ac54-c24f945cc204\" (UID: \"be4b2143-e746-45b5-ac54-c24f945cc204\") "
May 20 14:07:54.230065 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.229880    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be4b2143-e746-45b5-ac54-c24f945cc204" (UID: "be4b2143-e746-45b5-ac54-c24f945cc204"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:54.230065 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.229962    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-home" (OuterVolumeSpecName: "home") pod "be4b2143-e746-45b5-ac54-c24f945cc204" (UID: "be4b2143-e746-45b5-ac54-c24f945cc204"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:54.230221 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.230067    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-model-cache" (OuterVolumeSpecName: "model-cache") pod "be4b2143-e746-45b5-ac54-c24f945cc204" (UID: "be4b2143-e746-45b5-ac54-c24f945cc204"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:54.230221 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.230185    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:07:54.230221 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.230202    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:07:54.230221 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.230212    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:07:54.231856 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.231833    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4b2143-e746-45b5-ac54-c24f945cc204-kube-api-access-5chcg" (OuterVolumeSpecName: "kube-api-access-5chcg") pod "be4b2143-e746-45b5-ac54-c24f945cc204" (UID: "be4b2143-e746-45b5-ac54-c24f945cc204"). InnerVolumeSpecName "kube-api-access-5chcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:07:54.231962 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.231935    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4b2143-e746-45b5-ac54-c24f945cc204-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be4b2143-e746-45b5-ac54-c24f945cc204" (UID: "be4b2143-e746-45b5-ac54-c24f945cc204"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:07:54.232075 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.232058    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-dshm" (OuterVolumeSpecName: "dshm") pod "be4b2143-e746-45b5-ac54-c24f945cc204" (UID: "be4b2143-e746-45b5-ac54-c24f945cc204"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:54.285624 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.285578    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be4b2143-e746-45b5-ac54-c24f945cc204" (UID: "be4b2143-e746-45b5-ac54-c24f945cc204"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:07:54.331662 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.331628    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be4b2143-e746-45b5-ac54-c24f945cc204-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:07:54.331662 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.331658    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:07:54.331822 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.331670    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be4b2143-e746-45b5-ac54-c24f945cc204-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:07:54.331822 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.331680    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5chcg\" (UniqueName: \"kubernetes.io/projected/be4b2143-e746-45b5-ac54-c24f945cc204-kube-api-access-5chcg\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:07:54.862106 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.862068    2550 generic.go:358] "Generic (PLEG): container finished" podID="be4b2143-e746-45b5-ac54-c24f945cc204" containerID="e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e" exitCode=0
May 20 14:07:54.862282 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.862134    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw" event={"ID":"be4b2143-e746-45b5-ac54-c24f945cc204","Type":"ContainerDied","Data":"e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e"}
May 20 14:07:54.862282 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.862137    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"
May 20 14:07:54.862282 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.862160    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw" event={"ID":"be4b2143-e746-45b5-ac54-c24f945cc204","Type":"ContainerDied","Data":"28e1452ca207c12711db24e81e8d39a8b79a6592e03bea4c028ca3b7cd5bd34f"}
May 20 14:07:54.862282 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.862189    2550 scope.go:117] "RemoveContainer" containerID="e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e"
May 20 14:07:54.870539 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.870517    2550 scope.go:117] "RemoveContainer" containerID="0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556"
May 20 14:07:54.880727 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.880696    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"]
May 20 14:07:54.885203 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.885167    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-684c9d6b7-t4hcw"]
May 20 14:07:54.932672 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.932654    2550 scope.go:117] "RemoveContainer" containerID="e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e"
May 20 14:07:54.932999 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:07:54.932981    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e\": container with ID starting with e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e not found: ID does not exist" containerID="e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e"
May 20 14:07:54.933055 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.933009    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e"} err="failed to get container status \"e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e\": rpc error: code = NotFound desc = could not find container \"e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e\": container with ID starting with e7c93eaff6785ea7e052f953744dcf68e56071539d56b943cf331b2eb662fa0e not found: ID does not exist"
May 20 14:07:54.933055 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.933028    2550 scope.go:117] "RemoveContainer" containerID="0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556"
May 20 14:07:54.933298 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:07:54.933278    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556\": container with ID starting with 0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556 not found: ID does not exist" containerID="0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556"
May 20 14:07:54.933383 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:54.933310    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556"} err="failed to get container status \"0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556\": rpc error: code = NotFound desc = could not find container \"0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556\": container with ID starting with 0f220f845540d5b4b3424da2011a44ace42605ea9c61493a1e3447785c46b556 not found: ID does not exist"
May 20 14:07:56.467563 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:07:56.467531    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be4b2143-e746-45b5-ac54-c24f945cc204" path="/var/lib/kubelet/pods/be4b2143-e746-45b5-ac54-c24f945cc204/volumes"
May 20 14:08:00.847972 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.847937    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"]
May 20 14:08:00.848457 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.848350    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be4b2143-e746-45b5-ac54-c24f945cc204" containerName="storage-initializer"
May 20 14:08:00.848457 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.848372    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="be4b2143-e746-45b5-ac54-c24f945cc204" containerName="storage-initializer"
May 20 14:08:00.848457 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.848400    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be4b2143-e746-45b5-ac54-c24f945cc204" containerName="main"
May 20 14:08:00.848457 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.848409    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="be4b2143-e746-45b5-ac54-c24f945cc204" containerName="main"
May 20 14:08:00.848712 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.848496    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="be4b2143-e746-45b5-ac54-c24f945cc204" containerName="main"
May 20 14:08:00.852051 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.852032    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:00.854740 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.854715    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 20 14:08:00.854854 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.854753    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 14:08:00.854854 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.854775    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
May 20 14:08:00.854854 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.854790    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 14:08:00.855015 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.854861    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qp5wk\""
May 20 14:08:00.862238 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.862207    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"]
May 20 14:08:00.987273 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.987228    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:00.987490 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.987287    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6b4n\" (UniqueName: \"kubernetes.io/projected/b8893c98-a71f-46cb-958e-264f071a8265-kube-api-access-q6b4n\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:00.987490 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.987318    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:00.987490 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.987361    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:00.987490 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.987412    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:00.987490 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.987475    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:00.987667 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:00.987506    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8893c98-a71f-46cb-958e-264f071a8265-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.088581 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.088541    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8893c98-a71f-46cb-958e-264f071a8265-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.088789 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.088675    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.088789 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.088715    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q6b4n\" (UniqueName: \"kubernetes.io/projected/b8893c98-a71f-46cb-958e-264f071a8265-kube-api-access-q6b4n\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.088789 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.088762    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.088789 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.088784    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.089006 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.088835    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.089006 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.088890    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.089114 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.089097    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.089181 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.089154    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.089277 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.089258    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.089350 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.089290    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.090934 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.090906    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.091097 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.091081    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8893c98-a71f-46cb-958e-264f071a8265-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.097662 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.097639    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6b4n\" (UniqueName: \"kubernetes.io/projected/b8893c98-a71f-46cb-958e-264f071a8265-kube-api-access-q6b4n\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.163219 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.163141    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:08:01.300680 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.300645    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"]
May 20 14:08:01.304401 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:08:01.304376    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8893c98_a71f_46cb_958e_264f071a8265.slice/crio-98c8bdf4cd2adeca57afa3b5ffa60aa90e4f1c42bf2d779f482cf50829bb389e WatchSource:0}: Error finding container 98c8bdf4cd2adeca57afa3b5ffa60aa90e4f1c42bf2d779f482cf50829bb389e: Status 404 returned error can't find the container with id 98c8bdf4cd2adeca57afa3b5ffa60aa90e4f1c42bf2d779f482cf50829bb389e
May 20 14:08:01.891966 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.891922    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" event={"ID":"b8893c98-a71f-46cb-958e-264f071a8265","Type":"ContainerStarted","Data":"140f23b41bd6502feffd67acd34d3eba5194c19dae7839778b654332734b6dfd"}
May 20 14:08:01.891966 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:01.891969    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" event={"ID":"b8893c98-a71f-46cb-958e-264f071a8265","Type":"ContainerStarted","Data":"98c8bdf4cd2adeca57afa3b5ffa60aa90e4f1c42bf2d779f482cf50829bb389e"}
May 20 14:08:05.906411 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:05.906369    2550 generic.go:358] "Generic (PLEG): container finished" podID="b8893c98-a71f-46cb-958e-264f071a8265" containerID="140f23b41bd6502feffd67acd34d3eba5194c19dae7839778b654332734b6dfd" exitCode=0
May 20 14:08:05.906815 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:05.906428    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" event={"ID":"b8893c98-a71f-46cb-958e-264f071a8265","Type":"ContainerDied","Data":"140f23b41bd6502feffd67acd34d3eba5194c19dae7839778b654332734b6dfd"}
May 20 14:08:53.084550 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:53.084507    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" event={"ID":"b8893c98-a71f-46cb-958e-264f071a8265","Type":"ContainerStarted","Data":"138d0a1de31318f2af799630cb245a0068d4b2e2d53c1d93b79779f518126acc"}
May 20 14:08:53.109796 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:53.109722    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" podStartSLOduration=6.928985894 podStartE2EDuration="53.109701533s" podCreationTimestamp="2026-05-20 14:08:00 +0000 UTC" firstStartedPulling="2026-05-20 14:08:05.907584038 +0000 UTC m=+1064.033967172" lastFinishedPulling="2026-05-20 14:08:52.088299678 +0000 UTC m=+1110.214682811" observedRunningTime="2026-05-20 14:08:53.108038626 +0000 UTC m=+1111.234421811" watchObservedRunningTime="2026-05-20 14:08:53.109701533 +0000 UTC m=+1111.236084687"
May 20 14:08:55.160102 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.160067    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"]
May 20 14:08:55.162976 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.162956    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.165869 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.165850    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
May 20 14:08:55.173787 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.173763    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"]
May 20 14:08:55.188763 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.188719    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.188933 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.188784    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-dshm\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.188933 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.188810    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9872\" (UniqueName: \"kubernetes.io/projected/5d604d43-3426-4137-95fc-20ed2f7859c1-kube-api-access-c9872\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.188933 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.188838    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d604d43-3426-4137-95fc-20ed2f7859c1-tls-certs\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.189042 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.188924    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.189042 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.188988    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-home\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.189042 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.189006    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-model-cache\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.289703 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.289669    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.289897 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.289719    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-dshm\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.289897 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.289739    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c9872\" (UniqueName: \"kubernetes.io/projected/5d604d43-3426-4137-95fc-20ed2f7859c1-kube-api-access-c9872\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.290024 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.289886    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d604d43-3426-4137-95fc-20ed2f7859c1-tls-certs\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.290024 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.289968    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.290128 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.290024    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-home\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.290201 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.290176    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-model-cache\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.290270 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.290250    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.290377 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.290330    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.290502 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.290427    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-home\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.290565 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.290523    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-model-cache\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.291948 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.291926    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-dshm\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.292249 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.292231    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d604d43-3426-4137-95fc-20ed2f7859c1-tls-certs\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.298716 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.298697    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9872\" (UniqueName: \"kubernetes.io/projected/5d604d43-3426-4137-95fc-20ed2f7859c1-kube-api-access-c9872\") pod \"precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.477159 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.477064    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:08:55.622413 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:55.622258    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"]
May 20 14:08:55.625511 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:08:55.625482    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d604d43_3426_4137_95fc_20ed2f7859c1.slice/crio-3296f130d2aa72ebf9bdf8cd4949b536ebb80b4c0f27cc63ac851e3fa9a8cb36 WatchSource:0}: Error finding container 3296f130d2aa72ebf9bdf8cd4949b536ebb80b4c0f27cc63ac851e3fa9a8cb36: Status 404 returned error can't find the container with id 3296f130d2aa72ebf9bdf8cd4949b536ebb80b4c0f27cc63ac851e3fa9a8cb36
May 20 14:08:56.097318 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:56.097267    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c" event={"ID":"5d604d43-3426-4137-95fc-20ed2f7859c1","Type":"ContainerStarted","Data":"c85942141985862787baef950efd95f400109fa160e1c82914f9c1445d1827e6"}
May 20 14:08:56.097318 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:08:56.097318    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c" event={"ID":"5d604d43-3426-4137-95fc-20ed2f7859c1","Type":"ContainerStarted","Data":"3296f130d2aa72ebf9bdf8cd4949b536ebb80b4c0f27cc63ac851e3fa9a8cb36"}
May 20 14:09:00.115186 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:00.115096    2550 generic.go:358] "Generic (PLEG): container finished" podID="5d604d43-3426-4137-95fc-20ed2f7859c1" containerID="c85942141985862787baef950efd95f400109fa160e1c82914f9c1445d1827e6" exitCode=0
May 20 14:09:00.115186 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:00.115174    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c" event={"ID":"5d604d43-3426-4137-95fc-20ed2f7859c1","Type":"ContainerDied","Data":"c85942141985862787baef950efd95f400109fa160e1c82914f9c1445d1827e6"}
May 20 14:09:01.120704 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:01.120660    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c" event={"ID":"5d604d43-3426-4137-95fc-20ed2f7859c1","Type":"ContainerStarted","Data":"5f4d4565bf4e679056af41d5b42a0dc08cccb457fcbd6736439ad36b2d701e79"}
May 20 14:09:01.142223 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:01.142158    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c" podStartSLOduration=6.142139005 podStartE2EDuration="6.142139005s" podCreationTimestamp="2026-05-20 14:08:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:09:01.140979201 +0000 UTC m=+1119.267362354" watchObservedRunningTime="2026-05-20 14:09:01.142139005 +0000 UTC m=+1119.268522159"
May 20 14:09:01.163936 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:01.163895    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:09:01.164106 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:01.163958    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:09:01.165739 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:01.165709    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 20 14:09:05.478080 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:05.478036    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:09:05.478080 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:05.478089    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:09:05.491202 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:05.491169    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:09:06.150732 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:06.150702    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:09:11.164464 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:11.164412    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 20 14:09:21.163831 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:21.163778    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 20 14:09:28.023063 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.023025    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"]
May 20 14:09:28.024019 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.023979    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c" podUID="5d604d43-3426-4137-95fc-20ed2f7859c1" containerName="main" containerID="cri-o://5f4d4565bf4e679056af41d5b42a0dc08cccb457fcbd6736439ad36b2d701e79" gracePeriod=30
May 20 14:09:28.227083 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.227050    2550 generic.go:358] "Generic (PLEG): container finished" podID="5d604d43-3426-4137-95fc-20ed2f7859c1" containerID="5f4d4565bf4e679056af41d5b42a0dc08cccb457fcbd6736439ad36b2d701e79" exitCode=0
May 20 14:09:28.227276 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.227112    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c" event={"ID":"5d604d43-3426-4137-95fc-20ed2f7859c1","Type":"ContainerDied","Data":"5f4d4565bf4e679056af41d5b42a0dc08cccb457fcbd6736439ad36b2d701e79"}
May 20 14:09:28.288259 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.288191    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:09:28.302312 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.302269    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-kserve-provision-location\") pod \"5d604d43-3426-4137-95fc-20ed2f7859c1\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") "
May 20 14:09:28.302484 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.302379    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-model-cache\") pod \"5d604d43-3426-4137-95fc-20ed2f7859c1\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") "
May 20 14:09:28.302484 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.302416    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-tmp-dir\") pod \"5d604d43-3426-4137-95fc-20ed2f7859c1\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") "
May 20 14:09:28.302484 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.302451    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-home\") pod \"5d604d43-3426-4137-95fc-20ed2f7859c1\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") "
May 20 14:09:28.302484 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.302478    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-dshm\") pod \"5d604d43-3426-4137-95fc-20ed2f7859c1\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") "
May 20 14:09:28.302715 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.302551    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d604d43-3426-4137-95fc-20ed2f7859c1-tls-certs\") pod \"5d604d43-3426-4137-95fc-20ed2f7859c1\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") "
May 20 14:09:28.302715 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.302586    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9872\" (UniqueName: \"kubernetes.io/projected/5d604d43-3426-4137-95fc-20ed2f7859c1-kube-api-access-c9872\") pod \"5d604d43-3426-4137-95fc-20ed2f7859c1\" (UID: \"5d604d43-3426-4137-95fc-20ed2f7859c1\") "
May 20 14:09:28.302864 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.302788    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d604d43-3426-4137-95fc-20ed2f7859c1" (UID: "5d604d43-3426-4137-95fc-20ed2f7859c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:28.302864 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.302850    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-home" (OuterVolumeSpecName: "home") pod "5d604d43-3426-4137-95fc-20ed2f7859c1" (UID: "5d604d43-3426-4137-95fc-20ed2f7859c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:28.303101 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.303077    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d604d43-3426-4137-95fc-20ed2f7859c1" (UID: "5d604d43-3426-4137-95fc-20ed2f7859c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:28.305058 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.305025    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-dshm" (OuterVolumeSpecName: "dshm") pod "5d604d43-3426-4137-95fc-20ed2f7859c1" (UID: "5d604d43-3426-4137-95fc-20ed2f7859c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:28.305193 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.305173    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d604d43-3426-4137-95fc-20ed2f7859c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d604d43-3426-4137-95fc-20ed2f7859c1" (UID: "5d604d43-3426-4137-95fc-20ed2f7859c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:09:28.305327 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.305307    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d604d43-3426-4137-95fc-20ed2f7859c1-kube-api-access-c9872" (OuterVolumeSpecName: "kube-api-access-c9872") pod "5d604d43-3426-4137-95fc-20ed2f7859c1" (UID: "5d604d43-3426-4137-95fc-20ed2f7859c1"). InnerVolumeSpecName "kube-api-access-c9872". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:09:28.361570 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.361523    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d604d43-3426-4137-95fc-20ed2f7859c1" (UID: "5d604d43-3426-4137-95fc-20ed2f7859c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:09:28.403608 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.403578    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:09:28.403608 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.403606    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:09:28.403608 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.403614    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:09:28.403828 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.403624    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:09:28.403828 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.403632    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d604d43-3426-4137-95fc-20ed2f7859c1-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:09:28.403828 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.403643    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9872\" (UniqueName: \"kubernetes.io/projected/5d604d43-3426-4137-95fc-20ed2f7859c1-kube-api-access-c9872\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:09:28.403828 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:28.403657    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d604d43-3426-4137-95fc-20ed2f7859c1-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:09:29.233360 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:29.233300    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c" event={"ID":"5d604d43-3426-4137-95fc-20ed2f7859c1","Type":"ContainerDied","Data":"3296f130d2aa72ebf9bdf8cd4949b536ebb80b4c0f27cc63ac851e3fa9a8cb36"}
May 20 14:09:29.233360 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:29.233358    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"
May 20 14:09:29.233819 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:29.233374    2550 scope.go:117] "RemoveContainer" containerID="5f4d4565bf4e679056af41d5b42a0dc08cccb457fcbd6736439ad36b2d701e79"
May 20 14:09:29.241800 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:29.241776    2550 scope.go:117] "RemoveContainer" containerID="c85942141985862787baef950efd95f400109fa160e1c82914f9c1445d1827e6"
May 20 14:09:29.253556 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:29.253511    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"]
May 20 14:09:29.255582 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:29.255556    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7cbfd5474f-6xw9c"]
May 20 14:09:30.467798 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:30.467763    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d604d43-3426-4137-95fc-20ed2f7859c1" path="/var/lib/kubelet/pods/5d604d43-3426-4137-95fc-20ed2f7859c1/volumes"
May 20 14:09:31.164022 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:31.163974    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 20 14:09:41.164569 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:41.164523    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 20 14:09:46.472134 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.472095    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"]
May 20 14:09:46.472747 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.472537    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d604d43-3426-4137-95fc-20ed2f7859c1" containerName="main"
May 20 14:09:46.472747 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.472555    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d604d43-3426-4137-95fc-20ed2f7859c1" containerName="main"
May 20 14:09:46.472747 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.472581    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d604d43-3426-4137-95fc-20ed2f7859c1" containerName="storage-initializer"
May 20 14:09:46.472747 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.472587    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d604d43-3426-4137-95fc-20ed2f7859c1" containerName="storage-initializer"
May 20 14:09:46.472747 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.472670    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="5d604d43-3426-4137-95fc-20ed2f7859c1" containerName="main"
May 20 14:09:46.476520 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.476496    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.479493 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.479468    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"conv-test-round-trip-kserve-self-signed-certs\""
May 20 14:09:46.487928 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.487896    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"]
May 20 14:09:46.575568 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.575525    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-dshm\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.575568 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.575573    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tmp-dir\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.575854 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.575682    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-model-cache\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.575854 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.575751    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kserve-provision-location\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.575854 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.575805    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-home\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.575854 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.575832    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tls-certs\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.576006 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.575870    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zldxx\" (UniqueName: \"kubernetes.io/projected/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kube-api-access-zldxx\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.676759 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.676718    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zldxx\" (UniqueName: \"kubernetes.io/projected/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kube-api-access-zldxx\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.676920 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.676774    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-dshm\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.676920 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.676797    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tmp-dir\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.676920 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.676835    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-model-cache\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.676920 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.676869    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kserve-provision-location\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.676920 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.676889    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-home\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.676920 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.676905    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tls-certs\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.677402 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.677373    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-home\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.677595 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.677389    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kserve-provision-location\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.677683 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.677427    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tmp-dir\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.677683 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.677456    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-model-cache\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.679187 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.679163    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-dshm\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.679494 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.679472    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tls-certs\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.686179 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.686142    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zldxx\" (UniqueName: \"kubernetes.io/projected/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kube-api-access-zldxx\") pod \"conv-test-round-trip-kserve-56766c79cf-wgb7x\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") " pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.789499 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.789453    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:09:46.927256 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:46.927227    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"]
May 20 14:09:46.929934 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:09:46.929902    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53287fa_42f9_4463_8ee7_0f2f5b1df50c.slice/crio-fbb86ccc4f821febc362c03a1e169e2b5b7174b024c3deac557eb9fc951fb3d6 WatchSource:0}: Error finding container fbb86ccc4f821febc362c03a1e169e2b5b7174b024c3deac557eb9fc951fb3d6: Status 404 returned error can't find the container with id fbb86ccc4f821febc362c03a1e169e2b5b7174b024c3deac557eb9fc951fb3d6
May 20 14:09:47.301296 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:47.301252    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x" event={"ID":"a53287fa-42f9-4463-8ee7-0f2f5b1df50c","Type":"ContainerStarted","Data":"9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2"}
May 20 14:09:47.301482 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:47.301301    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x" event={"ID":"a53287fa-42f9-4463-8ee7-0f2f5b1df50c","Type":"ContainerStarted","Data":"fbb86ccc4f821febc362c03a1e169e2b5b7174b024c3deac557eb9fc951fb3d6"}
May 20 14:09:51.164066 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:51.164019    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 20 14:09:51.320794 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:51.320761    2550 generic.go:358] "Generic (PLEG): container finished" podID="a53287fa-42f9-4463-8ee7-0f2f5b1df50c" containerID="9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2" exitCode=0
May 20 14:09:51.320962 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:51.320808    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x" event={"ID":"a53287fa-42f9-4463-8ee7-0f2f5b1df50c","Type":"ContainerDied","Data":"9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2"}
May 20 14:09:52.268108 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.268069    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"]
May 20 14:09:52.270744 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.270723    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.273491 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.273463    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 20 14:09:52.283425 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.283385    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"]
May 20 14:09:52.326484 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.326445    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x" event={"ID":"a53287fa-42f9-4463-8ee7-0f2f5b1df50c","Type":"ContainerStarted","Data":"6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1"}
May 20 14:09:52.348850 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.348786    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x" podStartSLOduration=6.348768811 podStartE2EDuration="6.348768811s" podCreationTimestamp="2026-05-20 14:09:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:09:52.346784958 +0000 UTC m=+1170.473168106" watchObservedRunningTime="2026-05-20 14:09:52.348768811 +0000 UTC m=+1170.475151964"
May 20 14:09:52.431292 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.431235    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-kserve-provision-location\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.431292 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.431292    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-home\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.431571 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.431312    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-tmp-dir\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.431571 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.431350    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kpx8\" (UniqueName: \"kubernetes.io/projected/f3a1db9e-a23f-4213-a656-3215cd46d384-kube-api-access-7kpx8\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.431571 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.431474    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a1db9e-a23f-4213-a656-3215cd46d384-tls-certs\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.431571 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.431522    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-dshm\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.431793 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.431598    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-model-cache\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.532947 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.532856    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a1db9e-a23f-4213-a656-3215cd46d384-tls-certs\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.532947 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.532915    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-dshm\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.533164 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.532981    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-model-cache\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.533164 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.533055    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-kserve-provision-location\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.533164 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.533101    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-home\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.533164 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.533125    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-tmp-dir\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.533164 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.533155    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7kpx8\" (UniqueName: \"kubernetes.io/projected/f3a1db9e-a23f-4213-a656-3215cd46d384-kube-api-access-7kpx8\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.533684 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.533657    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-kserve-provision-location\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.533871 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.533785    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-model-cache\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.533984 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.533867    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-tmp-dir\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.534513 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.534489    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-home\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.536172 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.536111    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a1db9e-a23f-4213-a656-3215cd46d384-tls-certs\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.537196 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.537173    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-dshm\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.548114 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.548083    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kpx8\" (UniqueName: \"kubernetes.io/projected/f3a1db9e-a23f-4213-a656-3215cd46d384-kube-api-access-7kpx8\") pod \"stop-feature-test-kserve-96dfd8dc5-2gs2q\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.585871 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.585833    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:09:52.764298 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:52.764254    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"]
May 20 14:09:52.767276 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:09:52.767245    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a1db9e_a23f_4213_a656_3215cd46d384.slice/crio-a9aea7ba6ca52d134a83eb41c92cece4691a5fb4997caefc1a6fb24ba18db0c2 WatchSource:0}: Error finding container a9aea7ba6ca52d134a83eb41c92cece4691a5fb4997caefc1a6fb24ba18db0c2: Status 404 returned error can't find the container with id a9aea7ba6ca52d134a83eb41c92cece4691a5fb4997caefc1a6fb24ba18db0c2
May 20 14:09:53.332308 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:53.332272    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" event={"ID":"f3a1db9e-a23f-4213-a656-3215cd46d384","Type":"ContainerStarted","Data":"0e40af86f629ef4dc05f1eb8d0330d106e6c0a1d558bd853a5d4671d243e8e16"}
May 20 14:09:53.332687 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:53.332317    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" event={"ID":"f3a1db9e-a23f-4213-a656-3215cd46d384","Type":"ContainerStarted","Data":"a9aea7ba6ca52d134a83eb41c92cece4691a5fb4997caefc1a6fb24ba18db0c2"}
May 20 14:09:55.363839 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:55.363738    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"]
May 20 14:09:55.364304 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:55.364153    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x" podUID="a53287fa-42f9-4463-8ee7-0f2f5b1df50c" containerName="main" containerID="cri-o://6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1" gracePeriod=30
May 20 14:09:56.790416 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:09:56.790376    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:10:01.164544 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:01.164495    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 20 14:10:11.164388 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:11.164304    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 20 14:10:15.337647 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:10:15.337604    2550 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a1db9e_a23f_4213_a656_3215cd46d384.slice/crio-conmon-0e40af86f629ef4dc05f1eb8d0330d106e6c0a1d558bd853a5d4671d243e8e16.scope\": RecentStats: unable to find data in memory cache]"
May 20 14:10:15.433855 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:15.433812    2550 generic.go:358] "Generic (PLEG): container finished" podID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerID="0e40af86f629ef4dc05f1eb8d0330d106e6c0a1d558bd853a5d4671d243e8e16" exitCode=0
May 20 14:10:15.434040 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:15.433863    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" event={"ID":"f3a1db9e-a23f-4213-a656-3215cd46d384","Type":"ContainerDied","Data":"0e40af86f629ef4dc05f1eb8d0330d106e6c0a1d558bd853a5d4671d243e8e16"}
May 20 14:10:16.440465 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:16.440424    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" event={"ID":"f3a1db9e-a23f-4213-a656-3215cd46d384","Type":"ContainerStarted","Data":"9ac66bfc53234b8f6f9b9cbeab7dfb4d9fb1736b8d73190ab06d69a317a0480d"}
May 20 14:10:16.466242 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:16.466182    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podStartSLOduration=24.466167692 podStartE2EDuration="24.466167692s" podCreationTimestamp="2026-05-20 14:09:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:10:16.463490726 +0000 UTC m=+1194.589873879" watchObservedRunningTime="2026-05-20 14:10:16.466167692 +0000 UTC m=+1194.592550849"
May 20 14:10:21.164186 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:21.164130    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
May 20 14:10:22.586600 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:22.586534    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:10:22.586600 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:22.586598    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:10:22.588164 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:22.588128    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 20 14:10:25.619919 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.619847    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-round-trip-kserve-56766c79cf-wgb7x_a53287fa-42f9-4463-8ee7-0f2f5b1df50c/main/0.log"
May 20 14:10:25.620309 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.620209    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:10:25.772101 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.772059    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zldxx\" (UniqueName: \"kubernetes.io/projected/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kube-api-access-zldxx\") pod \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") "
May 20 14:10:25.772296 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.772123    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-home\") pod \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") "
May 20 14:10:25.772296 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.772159    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kserve-provision-location\") pod \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") "
May 20 14:10:25.772296 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.772230    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-dshm\") pod \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") "
May 20 14:10:25.772296 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.772284    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tls-certs\") pod \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") "
May 20 14:10:25.772562 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.772373    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-model-cache\") pod \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") "
May 20 14:10:25.772562 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.772430    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tmp-dir\") pod \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\" (UID: \"a53287fa-42f9-4463-8ee7-0f2f5b1df50c\") "
May 20 14:10:25.772562 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.772424    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-home" (OuterVolumeSpecName: "home") pod "a53287fa-42f9-4463-8ee7-0f2f5b1df50c" (UID: "a53287fa-42f9-4463-8ee7-0f2f5b1df50c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:10:25.772713 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.772650    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a53287fa-42f9-4463-8ee7-0f2f5b1df50c" (UID: "a53287fa-42f9-4463-8ee7-0f2f5b1df50c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:10:25.772713 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.772693    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:10:25.772977 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.772953    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a53287fa-42f9-4463-8ee7-0f2f5b1df50c" (UID: "a53287fa-42f9-4463-8ee7-0f2f5b1df50c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:10:25.774520 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.774492    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-dshm" (OuterVolumeSpecName: "dshm") pod "a53287fa-42f9-4463-8ee7-0f2f5b1df50c" (UID: "a53287fa-42f9-4463-8ee7-0f2f5b1df50c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:10:25.775016 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.774986    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a53287fa-42f9-4463-8ee7-0f2f5b1df50c" (UID: "a53287fa-42f9-4463-8ee7-0f2f5b1df50c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:10:25.775122 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.775054    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kube-api-access-zldxx" (OuterVolumeSpecName: "kube-api-access-zldxx") pod "a53287fa-42f9-4463-8ee7-0f2f5b1df50c" (UID: "a53287fa-42f9-4463-8ee7-0f2f5b1df50c"). InnerVolumeSpecName "kube-api-access-zldxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:10:25.840862 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.840814    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a53287fa-42f9-4463-8ee7-0f2f5b1df50c" (UID: "a53287fa-42f9-4463-8ee7-0f2f5b1df50c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:10:25.873243 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.873153    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:10:25.873243 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.873189    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:10:25.873243 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.873202    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:10:25.873243 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.873214    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zldxx\" (UniqueName: \"kubernetes.io/projected/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kube-api-access-zldxx\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:10:25.873243 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.873226    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:10:25.873243 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:25.873239    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53287fa-42f9-4463-8ee7-0f2f5b1df50c-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:10:26.478354 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.478301    2550 generic.go:358] "Generic (PLEG): container finished" podID="a53287fa-42f9-4463-8ee7-0f2f5b1df50c" containerID="6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1" exitCode=137
May 20 14:10:26.478551 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.478438    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x" event={"ID":"a53287fa-42f9-4463-8ee7-0f2f5b1df50c","Type":"ContainerDied","Data":"6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1"}
May 20 14:10:26.478551 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.478473    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x" event={"ID":"a53287fa-42f9-4463-8ee7-0f2f5b1df50c","Type":"ContainerDied","Data":"fbb86ccc4f821febc362c03a1e169e2b5b7174b024c3deac557eb9fc951fb3d6"}
May 20 14:10:26.478551 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.478497    2550 scope.go:117] "RemoveContainer" containerID="6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1"
May 20 14:10:26.478758 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.478665    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"
May 20 14:10:26.487943 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.487928    2550 scope.go:117] "RemoveContainer" containerID="9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2"
May 20 14:10:26.497244 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.497209    2550 scope.go:117] "RemoveContainer" containerID="6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1"
May 20 14:10:26.497534 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:10:26.497512    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1\": container with ID starting with 6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1 not found: ID does not exist" containerID="6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1"
May 20 14:10:26.497616 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.497541    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1"} err="failed to get container status \"6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1\": rpc error: code = NotFound desc = could not find container \"6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1\": container with ID starting with 6132b426964f2143d887fe6194cbd2a20f3b50bb7c8b0fa3bfa400182bcf59a1 not found: ID does not exist"
May 20 14:10:26.497616 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.497560    2550 scope.go:117] "RemoveContainer" containerID="9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2"
May 20 14:10:26.497811 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:10:26.497796    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2\": container with ID starting with 9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2 not found: ID does not exist" containerID="9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2"
May 20 14:10:26.497875 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.497814    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2"} err="failed to get container status \"9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2\": rpc error: code = NotFound desc = could not find container \"9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2\": container with ID starting with 9cfb2e9249cd02cb8f69bf5b64ce3c07eb76f98a0672174b7c9ea222bdb4b6f2 not found: ID does not exist"
May 20 14:10:26.514913 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.514872    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"]
May 20 14:10:26.519254 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:26.519225    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/conv-test-round-trip-kserve-56766c79cf-wgb7x"]
May 20 14:10:28.467177 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:28.467142    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53287fa-42f9-4463-8ee7-0f2f5b1df50c" path="/var/lib/kubelet/pods/a53287fa-42f9-4463-8ee7-0f2f5b1df50c/volumes"
May 20 14:10:31.174644 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:31.174610    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:10:31.182716 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:31.182684    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:10:32.587269 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:32.587221    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 20 14:10:37.456614 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:37.456569    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"]
May 20 14:10:37.457095 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:37.456962    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main" containerID="cri-o://138d0a1de31318f2af799630cb245a0068d4b2e2d53c1d93b79779f518126acc" gracePeriod=30
May 20 14:10:42.587057 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:42.587010    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 20 14:10:51.981060 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:51.981020    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"]
May 20 14:10:51.981595 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:51.981575    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53287fa-42f9-4463-8ee7-0f2f5b1df50c" containerName="storage-initializer"
May 20 14:10:51.981641 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:51.981599    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53287fa-42f9-4463-8ee7-0f2f5b1df50c" containerName="storage-initializer"
May 20 14:10:51.981641 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:51.981630    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53287fa-42f9-4463-8ee7-0f2f5b1df50c" containerName="main"
May 20 14:10:51.981641 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:51.981639    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53287fa-42f9-4463-8ee7-0f2f5b1df50c" containerName="main"
May 20 14:10:51.981751 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:51.981729    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="a53287fa-42f9-4463-8ee7-0f2f5b1df50c" containerName="main"
May 20 14:10:51.984255 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:51.984235    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:51.987145 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:51.987119    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 20 14:10:51.994935 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:51.994910    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"]
May 20 14:10:52.110992 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.110948    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-dshm\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.110992 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.110999    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-tmp-dir\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.111256 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.111026    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-model-cache\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.111256 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.111090    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29rlb\" (UniqueName: \"kubernetes.io/projected/d84ed730-6565-49c0-97f2-1b21e25d9738-kube-api-access-29rlb\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.111256 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.111142    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84ed730-6565-49c0-97f2-1b21e25d9738-tls-certs\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.111256 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.111194    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-home\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.111443 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.111256    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.212490 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.212436    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-dshm\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.212490 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.212484    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-tmp-dir\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.212757 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.212504    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-model-cache\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.212757 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.212532    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29rlb\" (UniqueName: \"kubernetes.io/projected/d84ed730-6565-49c0-97f2-1b21e25d9738-kube-api-access-29rlb\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.212757 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.212653    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84ed730-6565-49c0-97f2-1b21e25d9738-tls-certs\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.212757 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.212721    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-home\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.212975 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.212772    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.212975 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.212931    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-tmp-dir\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.213127 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.213100    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.213191 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.213115    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-home\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.213247 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.213234    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-model-cache\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.215289 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.215259    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-dshm\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.215460 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.215329    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84ed730-6565-49c0-97f2-1b21e25d9738-tls-certs\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.221756 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.221726    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29rlb\" (UniqueName: \"kubernetes.io/projected/d84ed730-6565-49c0-97f2-1b21e25d9738-kube-api-access-29rlb\") pod \"custom-route-timeout-test-kserve-6bb559dc87-6tns4\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.296184 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.296152    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:10:52.460381 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.460306    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"]
May 20 14:10:52.463865 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:10:52.463819    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84ed730_6565_49c0_97f2_1b21e25d9738.slice/crio-112cf3e9eec451d6e925f0558a541ecedd37862422b8fabd20da69af0ae292f9 WatchSource:0}: Error finding container 112cf3e9eec451d6e925f0558a541ecedd37862422b8fabd20da69af0ae292f9: Status 404 returned error can't find the container with id 112cf3e9eec451d6e925f0558a541ecedd37862422b8fabd20da69af0ae292f9
May 20 14:10:52.466435 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.466278    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:10:52.572266 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.572164    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" event={"ID":"d84ed730-6565-49c0-97f2-1b21e25d9738","Type":"ContainerStarted","Data":"4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41"}
May 20 14:10:52.572266 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.572207    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" event={"ID":"d84ed730-6565-49c0-97f2-1b21e25d9738","Type":"ContainerStarted","Data":"112cf3e9eec451d6e925f0558a541ecedd37862422b8fabd20da69af0ae292f9"}
May 20 14:10:52.586723 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:52.586678    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 20 14:10:57.592599 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:57.592568    2550 generic.go:358] "Generic (PLEG): container finished" podID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerID="4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41" exitCode=0
May 20 14:10:57.592980 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:57.592636    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" event={"ID":"d84ed730-6565-49c0-97f2-1b21e25d9738","Type":"ContainerDied","Data":"4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41"}
May 20 14:10:58.597783 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:58.597751    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" event={"ID":"d84ed730-6565-49c0-97f2-1b21e25d9738","Type":"ContainerStarted","Data":"bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193"}
May 20 14:10:58.620878 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:10:58.620814    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podStartSLOduration=7.620791495 podStartE2EDuration="7.620791495s" podCreationTimestamp="2026-05-20 14:10:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:10:58.619220459 +0000 UTC m=+1236.745603636" watchObservedRunningTime="2026-05-20 14:10:58.620791495 +0000 UTC m=+1236.747174649"
May 20 14:11:02.297112 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:02.297066    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:11:02.297112 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:02.297122    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:11:02.298560 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:02.298528    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
May 20 14:11:02.587254 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:02.587158    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 20 14:11:07.632460 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.632427    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t_b8893c98-a71f-46cb-958e-264f071a8265/main/0.log"
May 20 14:11:07.632873 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.632838    2550 generic.go:358] "Generic (PLEG): container finished" podID="b8893c98-a71f-46cb-958e-264f071a8265" containerID="138d0a1de31318f2af799630cb245a0068d4b2e2d53c1d93b79779f518126acc" exitCode=137
May 20 14:11:07.632980 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.632956    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" event={"ID":"b8893c98-a71f-46cb-958e-264f071a8265","Type":"ContainerDied","Data":"138d0a1de31318f2af799630cb245a0068d4b2e2d53c1d93b79779f518126acc"}
May 20 14:11:07.746095 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.746059    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t_b8893c98-a71f-46cb-958e-264f071a8265/main/0.log"
May 20 14:11:07.746524 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.746504    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:11:07.865706 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.865669    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-tmp-dir\") pod \"b8893c98-a71f-46cb-958e-264f071a8265\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") "
May 20 14:11:07.865895 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.865770    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-model-cache\") pod \"b8893c98-a71f-46cb-958e-264f071a8265\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") "
May 20 14:11:07.865895 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.865810    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6b4n\" (UniqueName: \"kubernetes.io/projected/b8893c98-a71f-46cb-958e-264f071a8265-kube-api-access-q6b4n\") pod \"b8893c98-a71f-46cb-958e-264f071a8265\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") "
May 20 14:11:07.865895 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.865886    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-kserve-provision-location\") pod \"b8893c98-a71f-46cb-958e-264f071a8265\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") "
May 20 14:11:07.866063 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.865918    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-dshm\") pod \"b8893c98-a71f-46cb-958e-264f071a8265\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") "
May 20 14:11:07.866063 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.865980    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-home\") pod \"b8893c98-a71f-46cb-958e-264f071a8265\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") "
May 20 14:11:07.866063 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.866015    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8893c98-a71f-46cb-958e-264f071a8265-tls-certs\") pod \"b8893c98-a71f-46cb-958e-264f071a8265\" (UID: \"b8893c98-a71f-46cb-958e-264f071a8265\") "
May 20 14:11:07.866201 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.866072    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8893c98-a71f-46cb-958e-264f071a8265" (UID: "b8893c98-a71f-46cb-958e-264f071a8265"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:11:07.866405 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.866372    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:11:07.866896 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.866863    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-home" (OuterVolumeSpecName: "home") pod "b8893c98-a71f-46cb-958e-264f071a8265" (UID: "b8893c98-a71f-46cb-958e-264f071a8265"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:11:07.869816 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.869783    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8893c98-a71f-46cb-958e-264f071a8265-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8893c98-a71f-46cb-958e-264f071a8265" (UID: "b8893c98-a71f-46cb-958e-264f071a8265"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:11:07.869928 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.869809    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8893c98-a71f-46cb-958e-264f071a8265-kube-api-access-q6b4n" (OuterVolumeSpecName: "kube-api-access-q6b4n") pod "b8893c98-a71f-46cb-958e-264f071a8265" (UID: "b8893c98-a71f-46cb-958e-264f071a8265"). InnerVolumeSpecName "kube-api-access-q6b4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:11:07.869928 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.869911    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-dshm" (OuterVolumeSpecName: "dshm") pod "b8893c98-a71f-46cb-958e-264f071a8265" (UID: "b8893c98-a71f-46cb-958e-264f071a8265"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:11:07.885412 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.885365    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8893c98-a71f-46cb-958e-264f071a8265" (UID: "b8893c98-a71f-46cb-958e-264f071a8265"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:11:07.933491 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.933448    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8893c98-a71f-46cb-958e-264f071a8265" (UID: "b8893c98-a71f-46cb-958e-264f071a8265"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:11:07.967210 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.967175    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:11:07.967210 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.967203    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8893c98-a71f-46cb-958e-264f071a8265-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:11:07.967210 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.967216    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:11:07.967516 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.967225    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6b4n\" (UniqueName: \"kubernetes.io/projected/b8893c98-a71f-46cb-958e-264f071a8265-kube-api-access-q6b4n\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:11:07.967516 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.967234    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:11:07.967516 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:07.967243    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8893c98-a71f-46cb-958e-264f071a8265-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:11:08.639737 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:08.639691    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t_b8893c98-a71f-46cb-958e-264f071a8265/main/0.log"
May 20 14:11:08.640263 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:08.640228    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t" event={"ID":"b8893c98-a71f-46cb-958e-264f071a8265","Type":"ContainerDied","Data":"98c8bdf4cd2adeca57afa3b5ffa60aa90e4f1c42bf2d779f482cf50829bb389e"}
May 20 14:11:08.640263 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:08.640263    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"
May 20 14:11:08.640441 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:08.640288    2550 scope.go:117] "RemoveContainer" containerID="138d0a1de31318f2af799630cb245a0068d4b2e2d53c1d93b79779f518126acc"
May 20 14:11:08.650298 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:08.650279    2550 scope.go:117] "RemoveContainer" containerID="140f23b41bd6502feffd67acd34d3eba5194c19dae7839778b654332734b6dfd"
May 20 14:11:08.663130 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:08.663091    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"]
May 20 14:11:08.665826 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:08.665794    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-5f4978c9bcksn8t"]
May 20 14:11:10.468605 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:10.468569    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8893c98-a71f-46cb-958e-264f071a8265" path="/var/lib/kubelet/pods/b8893c98-a71f-46cb-958e-264f071a8265/volumes"
May 20 14:11:12.296997 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:12.296904    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
May 20 14:11:12.586951 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:12.586855    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 20 14:11:22.297232 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:22.297181    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
May 20 14:11:22.587312 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:22.587214    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 20 14:11:32.297028 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:32.296982    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
May 20 14:11:32.587440 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:32.587319    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 20 14:11:42.297472 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:42.297422    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
May 20 14:11:42.586750 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:42.586657    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 20 14:11:52.297429 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:52.297377    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
May 20 14:11:52.587816 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:11:52.587664    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
May 20 14:12:02.296640 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:02.296592    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
May 20 14:12:02.597496 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:02.597405    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:12:02.606494 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:02.606466    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:12:03.638758 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:03.638722    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"]
May 20 14:12:03.852727 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:03.852557    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main" containerID="cri-o://9ac66bfc53234b8f6f9b9cbeab7dfb4d9fb1736b8d73190ab06d69a317a0480d" gracePeriod=30
May 20 14:12:12.297688 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:12.297639    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
May 20 14:12:21.954203 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:21.954167    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"]
May 20 14:12:21.958134 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:21.954558    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="storage-initializer"
May 20 14:12:21.958134 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:21.954571    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="storage-initializer"
May 20 14:12:21.958134 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:21.954582    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main"
May 20 14:12:21.958134 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:21.954588    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main"
May 20 14:12:21.958134 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:21.954660    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8893c98-a71f-46cb-958e-264f071a8265" containerName="main"
May 20 14:12:21.959422 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:21.959388    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:21.971525 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:21.971493    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"]
May 20 14:12:22.075756 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.075706    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-tmp-dir\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.075756 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.075751    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc64z\" (UniqueName: \"kubernetes.io/projected/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-kube-api-access-dc64z\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.076036 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.075779    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-home\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.076036 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.075833    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-model-cache\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.076036 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.075868    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-kserve-provision-location\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.076036 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.075898    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-dshm\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.076036 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.075999    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-tls-certs\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.176748 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.176702    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-kserve-provision-location\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.176748 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.176749    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-dshm\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.176992 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.176787    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-tls-certs\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.176992 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.176818    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-tmp-dir\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.176992 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.176837    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dc64z\" (UniqueName: \"kubernetes.io/projected/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-kube-api-access-dc64z\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.176992 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.176861    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-home\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.176992 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.176909    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-model-cache\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.177267 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.177205    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-kserve-provision-location\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.177328 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.177284    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-model-cache\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.177515 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.177464    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-tmp-dir\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.177515 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.177499    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-home\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.179319 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.179285    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-dshm\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.179441 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.179321    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-tls-certs\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.189873 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.189847    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc64z\" (UniqueName: \"kubernetes.io/projected/4dfc6c23-ee5b-4734-9325-bac9b9815fe2-kube-api-access-dc64z\") pod \"stop-feature-test-kserve-96dfd8dc5-2rrzb\" (UID: \"4dfc6c23-ee5b-4734-9325-bac9b9815fe2\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.273174 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.273137    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"
May 20 14:12:22.297357 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.297299    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
May 20 14:12:22.420888 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.420852    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb"]
May 20 14:12:22.425560 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:12:22.425534    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfc6c23_ee5b_4734_9325_bac9b9815fe2.slice/crio-0bc9ad764e1f3b26c90d51059a053aed2bbe4f4949badcf22f6f32966885f3f2 WatchSource:0}: Error finding container 0bc9ad764e1f3b26c90d51059a053aed2bbe4f4949badcf22f6f32966885f3f2: Status 404 returned error can't find the container with id 0bc9ad764e1f3b26c90d51059a053aed2bbe4f4949badcf22f6f32966885f3f2
May 20 14:12:22.926686 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.926642    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb" event={"ID":"4dfc6c23-ee5b-4734-9325-bac9b9815fe2","Type":"ContainerStarted","Data":"74ff65a4b243226e15a0930f2bfa3a7be6ae9ba00aecbdbaeda9a9247bc23718"}
May 20 14:12:22.926875 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:22.926693    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2rrzb" event={"ID":"4dfc6c23-ee5b-4734-9325-bac9b9815fe2","Type":"ContainerStarted","Data":"0bc9ad764e1f3b26c90d51059a053aed2bbe4f4949badcf22f6f32966885f3f2"}
May 20 14:12:32.296750 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:32.296692    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
May 20 14:12:33.970611 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:33.970586    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2gs2q_f3a1db9e-a23f-4213-a656-3215cd46d384/main/0.log"
May 20 14:12:33.971019 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:33.970938    2550 generic.go:358] "Generic (PLEG): container finished" podID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerID="9ac66bfc53234b8f6f9b9cbeab7dfb4d9fb1736b8d73190ab06d69a317a0480d" exitCode=137
May 20 14:12:33.971077 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:33.971010    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" event={"ID":"f3a1db9e-a23f-4213-a656-3215cd46d384","Type":"ContainerDied","Data":"9ac66bfc53234b8f6f9b9cbeab7dfb4d9fb1736b8d73190ab06d69a317a0480d"}
May 20 14:12:34.104097 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.104070    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2gs2q_f3a1db9e-a23f-4213-a656-3215cd46d384/main/0.log"
May 20 14:12:34.104481 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.104467    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:12:34.195602 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.195515    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-dshm\") pod \"f3a1db9e-a23f-4213-a656-3215cd46d384\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") "
May 20 14:12:34.195602 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.195562    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-model-cache\") pod \"f3a1db9e-a23f-4213-a656-3215cd46d384\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") "
May 20 14:12:34.195828 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.195610    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a1db9e-a23f-4213-a656-3215cd46d384-tls-certs\") pod \"f3a1db9e-a23f-4213-a656-3215cd46d384\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") "
May 20 14:12:34.195828 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.195664    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-tmp-dir\") pod \"f3a1db9e-a23f-4213-a656-3215cd46d384\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") "
May 20 14:12:34.195828 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.195731    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-kserve-provision-location\") pod \"f3a1db9e-a23f-4213-a656-3215cd46d384\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") "
May 20 14:12:34.195828 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.195768    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kpx8\" (UniqueName: \"kubernetes.io/projected/f3a1db9e-a23f-4213-a656-3215cd46d384-kube-api-access-7kpx8\") pod \"f3a1db9e-a23f-4213-a656-3215cd46d384\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") "
May 20 14:12:34.195828 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.195800    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-home\") pod \"f3a1db9e-a23f-4213-a656-3215cd46d384\" (UID: \"f3a1db9e-a23f-4213-a656-3215cd46d384\") "
May 20 14:12:34.196076 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.195862    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3a1db9e-a23f-4213-a656-3215cd46d384" (UID: "f3a1db9e-a23f-4213-a656-3215cd46d384"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:34.196128 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.196073    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:12:34.196530 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.196501    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-home" (OuterVolumeSpecName: "home") pod "f3a1db9e-a23f-4213-a656-3215cd46d384" (UID: "f3a1db9e-a23f-4213-a656-3215cd46d384"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:34.197834 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.197798    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-dshm" (OuterVolumeSpecName: "dshm") pod "f3a1db9e-a23f-4213-a656-3215cd46d384" (UID: "f3a1db9e-a23f-4213-a656-3215cd46d384"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:34.197936 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.197896    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a1db9e-a23f-4213-a656-3215cd46d384-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3a1db9e-a23f-4213-a656-3215cd46d384" (UID: "f3a1db9e-a23f-4213-a656-3215cd46d384"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:12:34.197986 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.197961    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a1db9e-a23f-4213-a656-3215cd46d384-kube-api-access-7kpx8" (OuterVolumeSpecName: "kube-api-access-7kpx8") pod "f3a1db9e-a23f-4213-a656-3215cd46d384" (UID: "f3a1db9e-a23f-4213-a656-3215cd46d384"). InnerVolumeSpecName "kube-api-access-7kpx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:12:34.214492 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.214449    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3a1db9e-a23f-4213-a656-3215cd46d384" (UID: "f3a1db9e-a23f-4213-a656-3215cd46d384"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:34.260752 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.260703    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3a1db9e-a23f-4213-a656-3215cd46d384" (UID: "f3a1db9e-a23f-4213-a656-3215cd46d384"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:12:34.297460 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.297421    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:12:34.297460 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.297454    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kpx8\" (UniqueName: \"kubernetes.io/projected/f3a1db9e-a23f-4213-a656-3215cd46d384-kube-api-access-7kpx8\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:12:34.297460 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.297464    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:12:34.297687 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.297475    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:12:34.297687 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.297484    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a1db9e-a23f-4213-a656-3215cd46d384-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:12:34.297687 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.297493    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a1db9e-a23f-4213-a656-3215cd46d384-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:12:34.976293 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.976263    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2gs2q_f3a1db9e-a23f-4213-a656-3215cd46d384/main/0.log"
May 20 14:12:34.976716 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.976659    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q" event={"ID":"f3a1db9e-a23f-4213-a656-3215cd46d384","Type":"ContainerDied","Data":"a9aea7ba6ca52d134a83eb41c92cece4691a5fb4997caefc1a6fb24ba18db0c2"}
May 20 14:12:34.976716 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.976704    2550 scope.go:117] "RemoveContainer" containerID="9ac66bfc53234b8f6f9b9cbeab7dfb4d9fb1736b8d73190ab06d69a317a0480d"
May 20 14:12:34.976797 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.976745    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"
May 20 14:12:34.985157 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.985138    2550 scope.go:117] "RemoveContainer" containerID="0e40af86f629ef4dc05f1eb8d0330d106e6c0a1d558bd853a5d4671d243e8e16"
May 20 14:12:34.999098 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:34.999057    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"]
May 20 14:12:35.005375 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:35.005318    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-96dfd8dc5-2gs2q"]
May 20 14:12:36.467086 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:36.467052    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" path="/var/lib/kubelet/pods/f3a1db9e-a23f-4213-a656-3215cd46d384/volumes"
May 20 14:12:42.306843 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:42.306757    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:12:42.314705 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:42.314677    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:12:48.207873 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:48.207839    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"]
May 20 14:12:48.208415 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:12:48.208204    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main" containerID="cri-o://bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193" gracePeriod=30
May 20 14:13:18.466380 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.466285    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-6bb559dc87-6tns4_d84ed730-6565-49c0-97f2-1b21e25d9738/main/0.log"
May 20 14:13:18.466811 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.466657    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:13:18.598268 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.598229    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-tmp-dir\") pod \"d84ed730-6565-49c0-97f2-1b21e25d9738\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") "
May 20 14:13:18.598483 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.598315    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84ed730-6565-49c0-97f2-1b21e25d9738-tls-certs\") pod \"d84ed730-6565-49c0-97f2-1b21e25d9738\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") "
May 20 14:13:18.598483 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.598371    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-kserve-provision-location\") pod \"d84ed730-6565-49c0-97f2-1b21e25d9738\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") "
May 20 14:13:18.598483 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.598407    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-dshm\") pod \"d84ed730-6565-49c0-97f2-1b21e25d9738\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") "
May 20 14:13:18.598483 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.598426    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29rlb\" (UniqueName: \"kubernetes.io/projected/d84ed730-6565-49c0-97f2-1b21e25d9738-kube-api-access-29rlb\") pod \"d84ed730-6565-49c0-97f2-1b21e25d9738\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") "
May 20 14:13:18.598483 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.598466    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-home\") pod \"d84ed730-6565-49c0-97f2-1b21e25d9738\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") "
May 20 14:13:18.598742 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.598520    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-model-cache\") pod \"d84ed730-6565-49c0-97f2-1b21e25d9738\" (UID: \"d84ed730-6565-49c0-97f2-1b21e25d9738\") "
May 20 14:13:18.601452 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.599687    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-model-cache" (OuterVolumeSpecName: "model-cache") pod "d84ed730-6565-49c0-97f2-1b21e25d9738" (UID: "d84ed730-6565-49c0-97f2-1b21e25d9738"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:13:18.601452 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.600180    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-home" (OuterVolumeSpecName: "home") pod "d84ed730-6565-49c0-97f2-1b21e25d9738" (UID: "d84ed730-6565-49c0-97f2-1b21e25d9738"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:13:18.610517 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.602098    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d84ed730-6565-49c0-97f2-1b21e25d9738-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d84ed730-6565-49c0-97f2-1b21e25d9738" (UID: "d84ed730-6565-49c0-97f2-1b21e25d9738"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:13:18.610517 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.602206    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84ed730-6565-49c0-97f2-1b21e25d9738-kube-api-access-29rlb" (OuterVolumeSpecName: "kube-api-access-29rlb") pod "d84ed730-6565-49c0-97f2-1b21e25d9738" (UID: "d84ed730-6565-49c0-97f2-1b21e25d9738"). InnerVolumeSpecName "kube-api-access-29rlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:13:18.611462 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.611417    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-dshm" (OuterVolumeSpecName: "dshm") pod "d84ed730-6565-49c0-97f2-1b21e25d9738" (UID: "d84ed730-6565-49c0-97f2-1b21e25d9738"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:13:18.616105 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.616059    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d84ed730-6565-49c0-97f2-1b21e25d9738" (UID: "d84ed730-6565-49c0-97f2-1b21e25d9738"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:13:18.659860 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.659778    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d84ed730-6565-49c0-97f2-1b21e25d9738" (UID: "d84ed730-6565-49c0-97f2-1b21e25d9738"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:13:18.699233 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.699196    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:13:18.699233 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.699230    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29rlb\" (UniqueName: \"kubernetes.io/projected/d84ed730-6565-49c0-97f2-1b21e25d9738-kube-api-access-29rlb\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:13:18.699233 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.699242    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:13:18.699503 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.699251    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:13:18.699503 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.699261    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:13:18.699503 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.699269    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d84ed730-6565-49c0-97f2-1b21e25d9738-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:13:18.699503 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:18.699277    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d84ed730-6565-49c0-97f2-1b21e25d9738-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:13:19.145527 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.145495    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-6bb559dc87-6tns4_d84ed730-6565-49c0-97f2-1b21e25d9738/main/0.log"
May 20 14:13:19.145806 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.145781    2550 generic.go:358] "Generic (PLEG): container finished" podID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerID="bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193" exitCode=137
May 20 14:13:19.145855 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.145834    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" event={"ID":"d84ed730-6565-49c0-97f2-1b21e25d9738","Type":"ContainerDied","Data":"bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193"}
May 20 14:13:19.145889 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.145868    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4" event={"ID":"d84ed730-6565-49c0-97f2-1b21e25d9738","Type":"ContainerDied","Data":"112cf3e9eec451d6e925f0558a541ecedd37862422b8fabd20da69af0ae292f9"}
May 20 14:13:19.145921 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.145891    2550 scope.go:117] "RemoveContainer" containerID="bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193"
May 20 14:13:19.145921 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.145897    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"
May 20 14:13:19.154671 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.154648    2550 scope.go:117] "RemoveContainer" containerID="4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41"
May 20 14:13:19.169955 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.169927    2550 scope.go:117] "RemoveContainer" containerID="bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193"
May 20 14:13:19.170301 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:13:19.170278    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193\": container with ID starting with bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193 not found: ID does not exist" containerID="bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193"
May 20 14:13:19.170415 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.170310    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193"} err="failed to get container status \"bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193\": rpc error: code = NotFound desc = could not find container \"bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193\": container with ID starting with bed2eef43a4635fefb96061eb927c996fb691e444044e7c8e34cd35a823a8193 not found: ID does not exist"
May 20 14:13:19.170415 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.170357    2550 scope.go:117] "RemoveContainer" containerID="4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41"
May 20 14:13:19.170661 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:13:19.170647    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41\": container with ID starting with 4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41 not found: ID does not exist" containerID="4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41"
May 20 14:13:19.170720 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.170664    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41"} err="failed to get container status \"4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41\": rpc error: code = NotFound desc = could not find container \"4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41\": container with ID starting with 4f3b903aaca5b1336d5b6c15db67d7a673eefe697a36980568694563b62a5e41 not found: ID does not exist"
May 20 14:13:19.170720 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.170693    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"]
May 20 14:13:19.173964 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:19.173932    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-6bb559dc87-6tns4"]
May 20 14:13:20.469740 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.469697    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" path="/var/lib/kubelet/pods/d84ed730-6565-49c0-97f2-1b21e25d9738/volumes"
May 20 14:13:20.481262 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.481232    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"]
May 20 14:13:20.481758 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.481739    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main"
May 20 14:13:20.481881 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.481760    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main"
May 20 14:13:20.481881 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.481772    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main"
May 20 14:13:20.481881 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.481780    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main"
May 20 14:13:20.481881 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.481794    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="storage-initializer"
May 20 14:13:20.481881 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.481802    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="storage-initializer"
May 20 14:13:20.481881 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.481816    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="storage-initializer"
May 20 14:13:20.481881 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.481824    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="storage-initializer"
May 20 14:13:20.482219 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.481908    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="f3a1db9e-a23f-4213-a656-3215cd46d384" containerName="main"
May 20 14:13:20.482219 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.481921    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="d84ed730-6565-49c0-97f2-1b21e25d9738" containerName="main"
May 20 14:13:20.485283 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.485262    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.488291 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.488273    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 20 14:13:20.496645 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.496606    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"]
May 20 14:13:20.616796 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.616756    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-dshm\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.616968 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.616852    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-kserve-provision-location\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.616968 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.616914    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a495bee-3bc3-4173-bc47-1b4d676ef619-tls-certs\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.616968 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.616949    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-home\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.617074 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.617038    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-model-cache\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.617109 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.617070    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-tmp-dir\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.617109 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.617096    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bbks\" (UniqueName: \"kubernetes.io/projected/4a495bee-3bc3-4173-bc47-1b4d676ef619-kube-api-access-4bbks\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.718546 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.718508    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-home\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.718717 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.718571    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-model-cache\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.718717 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.718587    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-tmp-dir\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.718717 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.718605    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4bbks\" (UniqueName: \"kubernetes.io/projected/4a495bee-3bc3-4173-bc47-1b4d676ef619-kube-api-access-4bbks\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.718717 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.718626    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-dshm\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.718717 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.718659    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-kserve-provision-location\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.719044 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.718817    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a495bee-3bc3-4173-bc47-1b4d676ef619-tls-certs\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.719044 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.718929    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-home\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.719044 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.718977    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-tmp-dir\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.719159 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.719065    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-kserve-provision-location\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.719201 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.719173    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-model-cache\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.721019 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.720950    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-dshm\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.721237 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.721197    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a495bee-3bc3-4173-bc47-1b4d676ef619-tls-certs\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.726962 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.726937    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bbks\" (UniqueName: \"kubernetes.io/projected/4a495bee-3bc3-4173-bc47-1b4d676ef619-kube-api-access-4bbks\") pod \"router-with-refs-test-kserve-fcd884cb7-x6msw\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.795201 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.795159    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:20.935823 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:20.935674    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"]
May 20 14:13:20.938615 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:13:20.938583    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a495bee_3bc3_4173_bc47_1b4d676ef619.slice/crio-0c811e3841c217bdb64744b1ce5200202b5a5400e7105dd75023ed0cf155140c WatchSource:0}: Error finding container 0c811e3841c217bdb64744b1ce5200202b5a5400e7105dd75023ed0cf155140c: Status 404 returned error can't find the container with id 0c811e3841c217bdb64744b1ce5200202b5a5400e7105dd75023ed0cf155140c
May 20 14:13:21.156375 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:21.156327    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" event={"ID":"4a495bee-3bc3-4173-bc47-1b4d676ef619","Type":"ContainerStarted","Data":"427cef4068fb6122a951a76a7ed694c0275799b78c2d34e169ad9a1d1bf96fa5"}
May 20 14:13:21.156375 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:21.156377    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" event={"ID":"4a495bee-3bc3-4173-bc47-1b4d676ef619","Type":"ContainerStarted","Data":"0c811e3841c217bdb64744b1ce5200202b5a5400e7105dd75023ed0cf155140c"}
May 20 14:13:26.177916 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:26.177884    2550 generic.go:358] "Generic (PLEG): container finished" podID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerID="427cef4068fb6122a951a76a7ed694c0275799b78c2d34e169ad9a1d1bf96fa5" exitCode=0
May 20 14:13:26.178331 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:26.177966    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" event={"ID":"4a495bee-3bc3-4173-bc47-1b4d676ef619","Type":"ContainerDied","Data":"427cef4068fb6122a951a76a7ed694c0275799b78c2d34e169ad9a1d1bf96fa5"}
May 20 14:13:27.182614 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:27.182582    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" event={"ID":"4a495bee-3bc3-4173-bc47-1b4d676ef619","Type":"ContainerStarted","Data":"181a8cbdec2629308b4ccccf1eca4f2dfa4b55e5f8387ec61b557869008e7d35"}
May 20 14:13:27.209120 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:27.209055    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" podStartSLOduration=7.209037826 podStartE2EDuration="7.209037826s" podCreationTimestamp="2026-05-20 14:13:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:13:27.208437289 +0000 UTC m=+1385.334820442" watchObservedRunningTime="2026-05-20 14:13:27.209037826 +0000 UTC m=+1385.335420980"
May 20 14:13:30.795610 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:30.795572    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:30.795610 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:30.795623    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:13:30.797371 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:30.797323    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 20 14:13:40.796540 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:40.796494    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 20 14:13:50.796074 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:13:50.796026    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 20 14:14:00.795928 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:14:00.795869    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 20 14:14:10.796183 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:14:10.796129    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 20 14:14:20.796304 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:14:20.796260    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 20 14:14:30.796264 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:14:30.796215    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 20 14:14:40.796540 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:14:40.796488    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 20 14:14:50.796387 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:14:50.796318    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": dial tcp 10.133.0.39:8000: connect: connection refused"
May 20 14:15:00.806264 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:00.806227    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:15:00.814789 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:00.814765    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:15:07.414963 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:07.414927    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"]
May 20 14:15:07.415465 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:07.415196    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main" containerID="cri-o://181a8cbdec2629308b4ccccf1eca4f2dfa4b55e5f8387ec61b557869008e7d35" gracePeriod=30
May 20 14:15:16.227774 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.227717    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"]
May 20 14:15:16.234708 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.234679    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"]
May 20 14:15:16.234898 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.234866    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.237916 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.237890    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 20 14:15:16.238050 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.237897    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-wvb49\""
May 20 14:15:16.238942 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.238920    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.244669 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.242838    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"]
May 20 14:15:16.245435 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.245412    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"]
May 20 14:15:16.248644 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.248616    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.248755 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.248664    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.248755 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.248691    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.248755 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.248736    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.248924 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.248763    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxw5h\" (UniqueName: \"kubernetes.io/projected/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kube-api-access-jxw5h\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.248924 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.248800    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.248924 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.248826    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfj5c\" (UniqueName: \"kubernetes.io/projected/2e665f73-e48f-481c-a32c-a89fd5732140-kube-api-access-lfj5c\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.248924 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.248858    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e665f73-e48f-481c-a32c-a89fd5732140-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.248924 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.248885    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.248924 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.248912    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.249233 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.249012    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.249233 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.249043    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.249233 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.249074    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.249233 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.249103    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.349948 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.349911    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.349948 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.349953    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.350206 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.349981    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.350206 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350005    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jxw5h\" (UniqueName: \"kubernetes.io/projected/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kube-api-access-jxw5h\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.350206 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350130    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.350206 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350171    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfj5c\" (UniqueName: \"kubernetes.io/projected/2e665f73-e48f-481c-a32c-a89fd5732140-kube-api-access-lfj5c\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.350478 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350213    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e665f73-e48f-481c-a32c-a89fd5732140-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.350478 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350245    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.350478 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350286    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.350478 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350383    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.350478 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350432    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.350478 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350470    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.350785 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350512    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.350785 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350569    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.350785 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350682    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.350785 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350711    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.351012 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350874    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.351012 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350943    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.351012 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.350396    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.351183 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.351029    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.351288 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.351264    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.351383 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.351298    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.352407 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.352380    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.352502 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.352433    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.352696 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.352679    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.352801 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.352784    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e665f73-e48f-481c-a32c-a89fd5732140-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.371541 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.371508    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxw5h\" (UniqueName: \"kubernetes.io/projected/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kube-api-access-jxw5h\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.371664 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.371556    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfj5c\" (UniqueName: \"kubernetes.io/projected/2e665f73-e48f-481c-a32c-a89fd5732140-kube-api-access-lfj5c\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.548365 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.548284    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:16.555322 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.555297    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:16.707833 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.707795    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"]
May 20 14:15:16.711573 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:15:16.711535    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d72dfee_d8b4_4fa3_8fc9_66673d88f76c.slice/crio-7ef49b1f9022fe3c83e8b98ac0215329f04da8baacd1539a6f2a2d26c9710131 WatchSource:0}: Error finding container 7ef49b1f9022fe3c83e8b98ac0215329f04da8baacd1539a6f2a2d26c9710131: Status 404 returned error can't find the container with id 7ef49b1f9022fe3c83e8b98ac0215329f04da8baacd1539a6f2a2d26c9710131
May 20 14:15:16.727699 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:16.727674    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"]
May 20 14:15:16.730064 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:15:16.730033    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e665f73_e48f_481c_a32c_a89fd5732140.slice/crio-5ba28a1bc8336a965ef2c9222a8aa6beeba98b2d52a259e7263ce593edad4e5e WatchSource:0}: Error finding container 5ba28a1bc8336a965ef2c9222a8aa6beeba98b2d52a259e7263ce593edad4e5e: Status 404 returned error can't find the container with id 5ba28a1bc8336a965ef2c9222a8aa6beeba98b2d52a259e7263ce593edad4e5e
May 20 14:15:17.564763 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:17.564718    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" event={"ID":"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c","Type":"ContainerStarted","Data":"7ef49b1f9022fe3c83e8b98ac0215329f04da8baacd1539a6f2a2d26c9710131"}
May 20 14:15:17.566378 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:17.566327    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" event={"ID":"2e665f73-e48f-481c-a32c-a89fd5732140","Type":"ContainerStarted","Data":"045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63"}
May 20 14:15:17.566511 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:17.566382    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" event={"ID":"2e665f73-e48f-481c-a32c-a89fd5732140","Type":"ContainerStarted","Data":"5ba28a1bc8336a965ef2c9222a8aa6beeba98b2d52a259e7263ce593edad4e5e"}
May 20 14:15:18.572028 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:18.571978    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" event={"ID":"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c","Type":"ContainerStarted","Data":"916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab"}
May 20 14:15:18.572463 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:18.572225    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:19.578085 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:19.578049    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" event={"ID":"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c","Type":"ContainerStarted","Data":"761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262"}
May 20 14:15:22.590433 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:22.590330    2550 generic.go:358] "Generic (PLEG): container finished" podID="2e665f73-e48f-481c-a32c-a89fd5732140" containerID="045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63" exitCode=0
May 20 14:15:22.590433 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:22.590411    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" event={"ID":"2e665f73-e48f-481c-a32c-a89fd5732140","Type":"ContainerDied","Data":"045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63"}
May 20 14:15:23.596523 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:23.596489    2550 generic.go:358] "Generic (PLEG): container finished" podID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerID="761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262" exitCode=0
May 20 14:15:23.597020 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:23.596554    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" event={"ID":"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c","Type":"ContainerDied","Data":"761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262"}
May 20 14:15:23.598387 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:23.598320    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" event={"ID":"2e665f73-e48f-481c-a32c-a89fd5732140","Type":"ContainerStarted","Data":"18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0"}
May 20 14:15:23.638551 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:23.638508    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podStartSLOduration=7.638492692 podStartE2EDuration="7.638492692s" podCreationTimestamp="2026-05-20 14:15:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:15:23.636624304 +0000 UTC m=+1501.763007458" watchObservedRunningTime="2026-05-20 14:15:23.638492692 +0000 UTC m=+1501.764875885"
May 20 14:15:24.604087 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:24.604051    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" event={"ID":"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c","Type":"ContainerStarted","Data":"7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e"}
May 20 14:15:24.629205 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:24.629142    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podStartSLOduration=7.572148182 podStartE2EDuration="8.629121252s" podCreationTimestamp="2026-05-20 14:15:16 +0000 UTC" firstStartedPulling="2026-05-20 14:15:16.713374934 +0000 UTC m=+1494.839758065" lastFinishedPulling="2026-05-20 14:15:17.770347991 +0000 UTC m=+1495.896731135" observedRunningTime="2026-05-20 14:15:24.628692321 +0000 UTC m=+1502.755075475" watchObservedRunningTime="2026-05-20 14:15:24.629121252 +0000 UTC m=+1502.755504406"
May 20 14:15:26.549291 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:26.549250    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:26.549291 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:26.549299    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:26.550646 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:26.550608    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:15:26.555838 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:26.555816    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:26.555932 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:26.555853    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:15:26.557031 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:26.557006    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:15:36.549235 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:36.549187    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:15:36.556053 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:36.556011    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:15:36.568890 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:36.568851    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:15:37.670576 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.670478    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-fcd884cb7-x6msw_4a495bee-3bc3-4173-bc47-1b4d676ef619/main/0.log"
May 20 14:15:37.671019 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.670923    2550 generic.go:358] "Generic (PLEG): container finished" podID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerID="181a8cbdec2629308b4ccccf1eca4f2dfa4b55e5f8387ec61b557869008e7d35" exitCode=137
May 20 14:15:37.671113 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.671090    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" event={"ID":"4a495bee-3bc3-4173-bc47-1b4d676ef619","Type":"ContainerDied","Data":"181a8cbdec2629308b4ccccf1eca4f2dfa4b55e5f8387ec61b557869008e7d35"}
May 20 14:15:37.729639 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.729609    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-fcd884cb7-x6msw_4a495bee-3bc3-4173-bc47-1b4d676ef619/main/0.log"
May 20 14:15:37.730054 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.730038    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:15:37.865709 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.865676    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-model-cache\") pod \"4a495bee-3bc3-4173-bc47-1b4d676ef619\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") "
May 20 14:15:37.865880 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.865745    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-kserve-provision-location\") pod \"4a495bee-3bc3-4173-bc47-1b4d676ef619\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") "
May 20 14:15:37.865880 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.865792    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a495bee-3bc3-4173-bc47-1b4d676ef619-tls-certs\") pod \"4a495bee-3bc3-4173-bc47-1b4d676ef619\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") "
May 20 14:15:37.865880 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.865826    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-tmp-dir\") pod \"4a495bee-3bc3-4173-bc47-1b4d676ef619\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") "
May 20 14:15:37.865880 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.865860    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bbks\" (UniqueName: \"kubernetes.io/projected/4a495bee-3bc3-4173-bc47-1b4d676ef619-kube-api-access-4bbks\") pod \"4a495bee-3bc3-4173-bc47-1b4d676ef619\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") "
May 20 14:15:37.866128 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.865961    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a495bee-3bc3-4173-bc47-1b4d676ef619" (UID: "4a495bee-3bc3-4173-bc47-1b4d676ef619"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:15:37.866128 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.866024    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-dshm\") pod \"4a495bee-3bc3-4173-bc47-1b4d676ef619\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") "
May 20 14:15:37.866128 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.866073    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-home\") pod \"4a495bee-3bc3-4173-bc47-1b4d676ef619\" (UID: \"4a495bee-3bc3-4173-bc47-1b4d676ef619\") "
May 20 14:15:37.866460 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.866430    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:15:37.866914 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.866879    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-home" (OuterVolumeSpecName: "home") pod "4a495bee-3bc3-4173-bc47-1b4d676ef619" (UID: "4a495bee-3bc3-4173-bc47-1b4d676ef619"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:15:37.868737 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.868707    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-dshm" (OuterVolumeSpecName: "dshm") pod "4a495bee-3bc3-4173-bc47-1b4d676ef619" (UID: "4a495bee-3bc3-4173-bc47-1b4d676ef619"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:15:37.868850 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.868710    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a495bee-3bc3-4173-bc47-1b4d676ef619-kube-api-access-4bbks" (OuterVolumeSpecName: "kube-api-access-4bbks") pod "4a495bee-3bc3-4173-bc47-1b4d676ef619" (UID: "4a495bee-3bc3-4173-bc47-1b4d676ef619"). InnerVolumeSpecName "kube-api-access-4bbks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:15:37.869026 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.868998    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a495bee-3bc3-4173-bc47-1b4d676ef619-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a495bee-3bc3-4173-bc47-1b4d676ef619" (UID: "4a495bee-3bc3-4173-bc47-1b4d676ef619"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:15:37.883085 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.883049    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a495bee-3bc3-4173-bc47-1b4d676ef619" (UID: "4a495bee-3bc3-4173-bc47-1b4d676ef619"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:15:37.944765 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.944648    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a495bee-3bc3-4173-bc47-1b4d676ef619" (UID: "4a495bee-3bc3-4173-bc47-1b4d676ef619"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:15:37.967692 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.967654    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:15:37.967692 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.967690    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a495bee-3bc3-4173-bc47-1b4d676ef619-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:15:37.967896 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.967706    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:15:37.967896 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.967716    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bbks\" (UniqueName: \"kubernetes.io/projected/4a495bee-3bc3-4173-bc47-1b4d676ef619-kube-api-access-4bbks\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:15:37.967896 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.967725    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:15:37.967896 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:37.967733    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a495bee-3bc3-4173-bc47-1b4d676ef619-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:15:38.676981 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:38.676954    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-fcd884cb7-x6msw_4a495bee-3bc3-4173-bc47-1b4d676ef619/main/0.log"
May 20 14:15:38.677406 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:38.677320    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw" event={"ID":"4a495bee-3bc3-4173-bc47-1b4d676ef619","Type":"ContainerDied","Data":"0c811e3841c217bdb64744b1ce5200202b5a5400e7105dd75023ed0cf155140c"}
May 20 14:15:38.677406 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:38.677371    2550 scope.go:117] "RemoveContainer" containerID="181a8cbdec2629308b4ccccf1eca4f2dfa4b55e5f8387ec61b557869008e7d35"
May 20 14:15:38.677509 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:38.677412    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"
May 20 14:15:38.686028 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:38.686001    2550 scope.go:117] "RemoveContainer" containerID="427cef4068fb6122a951a76a7ed694c0275799b78c2d34e169ad9a1d1bf96fa5"
May 20 14:15:38.700835 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:38.700808    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"]
May 20 14:15:38.706250 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:38.706226    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-fcd884cb7-x6msw"]
May 20 14:15:40.468660 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:40.468625    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" path="/var/lib/kubelet/pods/4a495bee-3bc3-4173-bc47-1b4d676ef619/volumes"
May 20 14:15:46.549677 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:46.549578    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:15:46.556510 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:46.556466    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:15:56.549839 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:56.549790    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:15:56.556300 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:15:56.556255    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:16:06.549145 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:06.549097    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:16:06.556380 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:06.556318    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:16:16.549385 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:16.549322    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:16:16.556076 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:16.556044    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:16:26.549046 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:26.548996    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:16:26.556483 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:26.556445    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:16:36.548841 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:36.548775    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:16:36.556552 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:36.556511    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:16:46.548707 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:46.548654    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:16:46.556228 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:46.556192    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:16:56.549459 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:56.549402    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:16:56.556552 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:16:56.556511    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:17:06.548941 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:06.548888    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:17:06.556633 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:06.556597    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:17:16.549143 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:16.549083    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:17:16.555737 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:16.555696    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:17:26.549718 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:26.549657    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:17:26.555959 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:26.555922    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:17:36.549124 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:36.549066    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8001/health\": dial tcp 10.133.0.40:8001: connect: connection refused"
May 20 14:17:36.556366 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:36.556309    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
May 20 14:17:46.558841 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:46.558808    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:17:46.567816 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:46.567783    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:17:46.573198 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:46.573173    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:17:46.576442 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:46.576427    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:17:55.376090 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:55.376056    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"]
May 20 14:17:55.376601 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:55.376358    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main" containerID="cri-o://18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0" gracePeriod=30
May 20 14:17:55.379906 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:55.379877    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"]
May 20 14:17:55.380457 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:17:55.380396    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main" containerID="cri-o://7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e" gracePeriod=30
May 20 14:18:02.799627 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.799591    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"]
May 20 14:18:02.800443 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.800420    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="storage-initializer"
May 20 14:18:02.800443 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.800446    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="storage-initializer"
May 20 14:18:02.800566 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.800460    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main"
May 20 14:18:02.800566 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.800470    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main"
May 20 14:18:02.800626 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.800584    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="4a495bee-3bc3-4173-bc47-1b4d676ef619" containerName="main"
May 20 14:18:02.804694 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.804668    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.807669 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.807636    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-tmfjr\""
May 20 14:18:02.808540 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.808518    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 20 14:18:02.817435 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.817400    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"]
May 20 14:18:02.821123 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.821098    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"]
May 20 14:18:02.821254 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.821220    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.830535 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.830504    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"]
May 20 14:18:02.846808 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.846771    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-dshm\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.846987 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.846810    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.846987 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.846844    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc900de-4093-4423-b190-b48c648a849e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.846987 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.846867    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.846987 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.846883    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.846987 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.846968    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfgl\" (UniqueName: \"kubernetes.io/projected/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kube-api-access-8mfgl\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.847185 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.846999    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.847185 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.847020    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.847185 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.847051    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.847185 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.847078    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-home\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.847185 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.847096    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.847185 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.847135    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc4jd\" (UniqueName: \"kubernetes.io/projected/ebc900de-4093-4423-b190-b48c648a849e-kube-api-access-kc4jd\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.847185 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.847166    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-model-cache\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.847456 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.847193    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.948705 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948665    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.948705 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948710    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.948965 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948738    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.948965 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948757    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-home\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.948965 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948773    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.948965 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948802    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kc4jd\" (UniqueName: \"kubernetes.io/projected/ebc900de-4093-4423-b190-b48c648a849e-kube-api-access-kc4jd\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.948965 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948833    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-model-cache\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.948965 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948859    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.948965 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948894    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-dshm\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.948965 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948923    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.948965 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948953    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc900de-4093-4423-b190-b48c648a849e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.949539 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.948979    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.949539 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.949002    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.949539 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.949090    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfgl\" (UniqueName: \"kubernetes.io/projected/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kube-api-access-8mfgl\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.949539 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.949117    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.949539 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.949187    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.949539 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.949212    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-home\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.949539 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.949234    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.949539 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.949256    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-model-cache\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.949539 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.949520    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.949993 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.949615    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.949993 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.949830    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.951524 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.951495    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.951670 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.951650    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.951815 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.951797    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-dshm\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.952061 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.952047    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc900de-4093-4423-b190-b48c648a849e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:02.957618 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.957595    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfgl\" (UniqueName: \"kubernetes.io/projected/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kube-api-access-8mfgl\") pod \"custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:02.957793 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:02.957775    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc4jd\" (UniqueName: \"kubernetes.io/projected/ebc900de-4093-4423-b190-b48c648a849e-kube-api-access-kc4jd\") pod \"custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:03.116692 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:03.116585    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:03.133028 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:03.132970    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:18:03.269445 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:03.269416    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"]
May 20 14:18:03.272640 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:18:03.272611    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d338f4_9c17_4a22_b0a6_e18884fbf43d.slice/crio-82841eb79d4a9386d88f887fd2e44342d1f3168807a9213badab3f2ce42e9287 WatchSource:0}: Error finding container 82841eb79d4a9386d88f887fd2e44342d1f3168807a9213badab3f2ce42e9287: Status 404 returned error can't find the container with id 82841eb79d4a9386d88f887fd2e44342d1f3168807a9213badab3f2ce42e9287
May 20 14:18:03.274481 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:03.274463    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:18:03.292043 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:03.291941    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"]
May 20 14:18:03.294690 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:18:03.294664    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc900de_4093_4423_b190_b48c648a849e.slice/crio-a96bf39060966a9789572f375e657c1f3557228287a70d216d64dcb7c2cf033f WatchSource:0}: Error finding container a96bf39060966a9789572f375e657c1f3557228287a70d216d64dcb7c2cf033f: Status 404 returned error can't find the container with id a96bf39060966a9789572f375e657c1f3557228287a70d216d64dcb7c2cf033f
May 20 14:18:04.209267 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:04.209224    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" event={"ID":"ebc900de-4093-4423-b190-b48c648a849e","Type":"ContainerStarted","Data":"e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa"}
May 20 14:18:04.209267 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:04.209264    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" event={"ID":"ebc900de-4093-4423-b190-b48c648a849e","Type":"ContainerStarted","Data":"a96bf39060966a9789572f375e657c1f3557228287a70d216d64dcb7c2cf033f"}
May 20 14:18:04.210563 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:04.210539    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" event={"ID":"45d338f4-9c17-4a22-b0a6-e18884fbf43d","Type":"ContainerStarted","Data":"5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef"}
May 20 14:18:04.210651 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:04.210573    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" event={"ID":"45d338f4-9c17-4a22-b0a6-e18884fbf43d","Type":"ContainerStarted","Data":"82841eb79d4a9386d88f887fd2e44342d1f3168807a9213badab3f2ce42e9287"}
May 20 14:18:04.210651 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:04.210642    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:05.216754 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:05.216715    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" event={"ID":"45d338f4-9c17-4a22-b0a6-e18884fbf43d","Type":"ContainerStarted","Data":"18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06"}
May 20 14:18:16.233236 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:16.233207    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:18:25.380953 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.380896    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="llm-d-routing-sidecar" containerID="cri-o://916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab" gracePeriod=2
May 20 14:18:25.688676 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.688650    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:18:25.778107 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.778077    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfj5c\" (UniqueName: \"kubernetes.io/projected/2e665f73-e48f-481c-a32c-a89fd5732140-kube-api-access-lfj5c\") pod \"2e665f73-e48f-481c-a32c-a89fd5732140\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") "
May 20 14:18:25.778280 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.778121    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-tmp-dir\") pod \"2e665f73-e48f-481c-a32c-a89fd5732140\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") "
May 20 14:18:25.778280 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.778239    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-model-cache\") pod \"2e665f73-e48f-481c-a32c-a89fd5732140\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") "
May 20 14:18:25.778280 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.778276    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-kserve-provision-location\") pod \"2e665f73-e48f-481c-a32c-a89fd5732140\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") "
May 20 14:18:25.778485 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.778313    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-home\") pod \"2e665f73-e48f-481c-a32c-a89fd5732140\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") "
May 20 14:18:25.778485 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.778381    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-dshm\") pod \"2e665f73-e48f-481c-a32c-a89fd5732140\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") "
May 20 14:18:25.778485 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.778416    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e665f73-e48f-481c-a32c-a89fd5732140-tls-certs\") pod \"2e665f73-e48f-481c-a32c-a89fd5732140\" (UID: \"2e665f73-e48f-481c-a32c-a89fd5732140\") "
May 20 14:18:25.778659 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.778515    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e665f73-e48f-481c-a32c-a89fd5732140" (UID: "2e665f73-e48f-481c-a32c-a89fd5732140"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:18:25.778963 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.778774    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:25.779191 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.779157    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-home" (OuterVolumeSpecName: "home") pod "2e665f73-e48f-481c-a32c-a89fd5732140" (UID: "2e665f73-e48f-481c-a32c-a89fd5732140"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:18:25.780902 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.780877    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e665f73-e48f-481c-a32c-a89fd5732140-kube-api-access-lfj5c" (OuterVolumeSpecName: "kube-api-access-lfj5c") pod "2e665f73-e48f-481c-a32c-a89fd5732140" (UID: "2e665f73-e48f-481c-a32c-a89fd5732140"). InnerVolumeSpecName "kube-api-access-lfj5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:18:25.781032 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.781013    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e665f73-e48f-481c-a32c-a89fd5732140-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e665f73-e48f-481c-a32c-a89fd5732140" (UID: "2e665f73-e48f-481c-a32c-a89fd5732140"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:18:25.781079 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.781045    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-dshm" (OuterVolumeSpecName: "dshm") pod "2e665f73-e48f-481c-a32c-a89fd5732140" (UID: "2e665f73-e48f-481c-a32c-a89fd5732140"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:18:25.790616 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.790586    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e665f73-e48f-481c-a32c-a89fd5732140" (UID: "2e665f73-e48f-481c-a32c-a89fd5732140"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:18:25.798567 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.798548    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57_8d72dfee-d8b4-4fa3-8fc9-66673d88f76c/main/0.log"
May 20 14:18:25.799231 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.799215    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:18:25.839084 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.839042    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e665f73-e48f-481c-a32c-a89fd5732140" (UID: "2e665f73-e48f-481c-a32c-a89fd5732140"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:18:25.880009 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.879974    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:25.880009 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.880004    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:25.880009 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.880014    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:25.880230 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.880022    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e665f73-e48f-481c-a32c-a89fd5732140-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:25.880230 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.880032    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfj5c\" (UniqueName: \"kubernetes.io/projected/2e665f73-e48f-481c-a32c-a89fd5732140-kube-api-access-lfj5c\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:25.880230 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.880041    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e665f73-e48f-481c-a32c-a89fd5732140-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:25.980941 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.980842    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxw5h\" (UniqueName: \"kubernetes.io/projected/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kube-api-access-jxw5h\") pod \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") "
May 20 14:18:25.980941 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.980887    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-dshm\") pod \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") "
May 20 14:18:25.980941 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.980907    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-home\") pod \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") "
May 20 14:18:25.981230 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.981005    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tmp-dir\") pod \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") "
May 20 14:18:25.981230 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.981085    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kserve-provision-location\") pod \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") "
May 20 14:18:25.981230 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.981118    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-model-cache\") pod \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") "
May 20 14:18:25.981230 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.981150    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tls-certs\") pod \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\" (UID: \"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c\") "
May 20 14:18:25.981490 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.981413    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" (UID: "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:18:25.981688 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.981663    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-home" (OuterVolumeSpecName: "home") pod "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" (UID: "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:18:25.983093 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.983064    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kube-api-access-jxw5h" (OuterVolumeSpecName: "kube-api-access-jxw5h") pod "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" (UID: "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c"). InnerVolumeSpecName "kube-api-access-jxw5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:18:25.983506 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.983481    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-dshm" (OuterVolumeSpecName: "dshm") pod "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" (UID: "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:18:25.983624 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.983604    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" (UID: "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:18:25.994774 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:25.994728    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" (UID: "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:18:26.036025 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.035962    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" (UID: "8d72dfee-d8b4-4fa3-8fc9-66673d88f76c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:18:26.082286 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.082247    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:26.082286 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.082282    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:26.082286 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.082292    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:26.082554 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.082304    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxw5h\" (UniqueName: \"kubernetes.io/projected/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-kube-api-access-jxw5h\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:26.082554 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.082318    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:26.082554 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.082328    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:26.082554 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.082362    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:18:26.292301 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.292263    2550 generic.go:358] "Generic (PLEG): container finished" podID="2e665f73-e48f-481c-a32c-a89fd5732140" containerID="18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0" exitCode=137
May 20 14:18:26.292501 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.292373    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"
May 20 14:18:26.292501 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.292374    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" event={"ID":"2e665f73-e48f-481c-a32c-a89fd5732140","Type":"ContainerDied","Data":"18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0"}
May 20 14:18:26.292501 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.292413    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk" event={"ID":"2e665f73-e48f-481c-a32c-a89fd5732140","Type":"ContainerDied","Data":"5ba28a1bc8336a965ef2c9222a8aa6beeba98b2d52a259e7263ce593edad4e5e"}
May 20 14:18:26.292501 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.292430    2550 scope.go:117] "RemoveContainer" containerID="18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0"
May 20 14:18:26.293700 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.293684    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57_8d72dfee-d8b4-4fa3-8fc9-66673d88f76c/main/0.log"
May 20 14:18:26.294355 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.294303    2550 generic.go:358] "Generic (PLEG): container finished" podID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerID="7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e" exitCode=137
May 20 14:18:26.294355 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.294324    2550 generic.go:358] "Generic (PLEG): container finished" podID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerID="916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab" exitCode=0
May 20 14:18:26.294473 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.294375    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" event={"ID":"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c","Type":"ContainerDied","Data":"7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e"}
May 20 14:18:26.294473 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.294428    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" event={"ID":"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c","Type":"ContainerDied","Data":"916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab"}
May 20 14:18:26.294473 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.294448    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57" event={"ID":"8d72dfee-d8b4-4fa3-8fc9-66673d88f76c","Type":"ContainerDied","Data":"7ef49b1f9022fe3c83e8b98ac0215329f04da8baacd1539a6f2a2d26c9710131"}
May 20 14:18:26.294473 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.294393    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"
May 20 14:18:26.301982 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.301887    2550 scope.go:117] "RemoveContainer" containerID="045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63"
May 20 14:18:26.313069 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.313051    2550 scope.go:117] "RemoveContainer" containerID="18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0"
May 20 14:18:26.313390 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:18:26.313358    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0\": container with ID starting with 18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0 not found: ID does not exist" containerID="18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0"
May 20 14:18:26.313513 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.313398    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0"} err="failed to get container status \"18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0\": rpc error: code = NotFound desc = could not find container \"18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0\": container with ID starting with 18409ca53f819864051045b51f3d3754f4dca262a20bc224a10b2a42c76875b0 not found: ID does not exist"
May 20 14:18:26.313513 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.313423    2550 scope.go:117] "RemoveContainer" containerID="045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63"
May 20 14:18:26.313916 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:18:26.313892    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63\": container with ID starting with 045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63 not found: ID does not exist" containerID="045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63"
May 20 14:18:26.313999 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.313924    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63"} err="failed to get container status \"045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63\": rpc error: code = NotFound desc = could not find container \"045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63\": container with ID starting with 045be8b25ffb6aeb15dec1c336026b0dc6f78e00450e5344974dca20cb758c63 not found: ID does not exist"
May 20 14:18:26.313999 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.313948    2550 scope.go:117] "RemoveContainer" containerID="7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e"
May 20 14:18:26.316469 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.316445    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"]
May 20 14:18:26.320895 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.320871    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6fcl8vk"]
May 20 14:18:26.324401 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.324379    2550 scope.go:117] "RemoveContainer" containerID="761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262"
May 20 14:18:26.333940 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.333907    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"]
May 20 14:18:26.335796 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.335773    2550 scope.go:117] "RemoveContainer" containerID="916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab"
May 20 14:18:26.336139 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.336122    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-f8dd46d4b-f8l57"]
May 20 14:18:26.342991 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.342970    2550 scope.go:117] "RemoveContainer" containerID="7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e"
May 20 14:18:26.343281 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:18:26.343262    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e\": container with ID starting with 7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e not found: ID does not exist" containerID="7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e"
May 20 14:18:26.343379 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.343291    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e"} err="failed to get container status \"7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e\": rpc error: code = NotFound desc = could not find container \"7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e\": container with ID starting with 7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e not found: ID does not exist"
May 20 14:18:26.343379 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.343311    2550 scope.go:117] "RemoveContainer" containerID="761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262"
May 20 14:18:26.343599 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:18:26.343583    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262\": container with ID starting with 761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262 not found: ID does not exist" containerID="761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262"
May 20 14:18:26.343650 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.343603    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262"} err="failed to get container status \"761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262\": rpc error: code = NotFound desc = could not find container \"761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262\": container with ID starting with 761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262 not found: ID does not exist"
May 20 14:18:26.343650 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.343619    2550 scope.go:117] "RemoveContainer" containerID="916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab"
May 20 14:18:26.343859 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:18:26.343844    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab\": container with ID starting with 916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab not found: ID does not exist" containerID="916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab"
May 20 14:18:26.343911 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.343863    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab"} err="failed to get container status \"916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab\": rpc error: code = NotFound desc = could not find container \"916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab\": container with ID starting with 916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab not found: ID does not exist"
May 20 14:18:26.343911 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.343876    2550 scope.go:117] "RemoveContainer" containerID="7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e"
May 20 14:18:26.344082 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.344068    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e"} err="failed to get container status \"7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e\": rpc error: code = NotFound desc = could not find container \"7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e\": container with ID starting with 7bdc8e7ab801e03792a9dabdfd70daf2217f22a70189c01cf842767145cbd23e not found: ID does not exist"
May 20 14:18:26.344124 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.344082    2550 scope.go:117] "RemoveContainer" containerID="761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262"
May 20 14:18:26.344267 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.344252    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262"} err="failed to get container status \"761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262\": rpc error: code = NotFound desc = could not find container \"761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262\": container with ID starting with 761643865a5257f4fde0536acf61b78264d648b1bf14a20cede365336a59f262 not found: ID does not exist"
May 20 14:18:26.344319 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.344266    2550 scope.go:117] "RemoveContainer" containerID="916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab"
May 20 14:18:26.344500 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.344481    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab"} err="failed to get container status \"916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab\": rpc error: code = NotFound desc = could not find container \"916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab\": container with ID starting with 916e4ac74814190bf9db3cf5407942989bf1c7676bd034e11b02772c00d1aeab not found: ID does not exist"
May 20 14:18:26.467056 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.467023    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" path="/var/lib/kubelet/pods/2e665f73-e48f-481c-a32c-a89fd5732140/volumes"
May 20 14:18:26.467501 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:26.467486    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" path="/var/lib/kubelet/pods/8d72dfee-d8b4-4fa3-8fc9-66673d88f76c/volumes"
May 20 14:18:59.414778 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:59.414735    2550 generic.go:358] "Generic (PLEG): container finished" podID="ebc900de-4093-4423-b190-b48c648a849e" containerID="e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa" exitCode=0
May 20 14:18:59.415172 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:18:59.414812    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" event={"ID":"ebc900de-4093-4423-b190-b48c648a849e","Type":"ContainerDied","Data":"e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa"}
May 20 14:19:00.420156 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:19:00.420126    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" event={"ID":"ebc900de-4093-4423-b190-b48c648a849e","Type":"ContainerStarted","Data":"0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464"}
May 20 14:19:00.446230 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:19:00.446167    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podStartSLOduration=58.44614945 podStartE2EDuration="58.44614945s" podCreationTimestamp="2026-05-20 14:18:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:19:00.44350952 +0000 UTC m=+1718.569892670" watchObservedRunningTime="2026-05-20 14:19:00.44614945 +0000 UTC m=+1718.572532604"
May 20 14:19:03.133932 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:19:03.133896    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:19:03.134368 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:19:03.133950    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:19:03.135285 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:19:03.135259    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 20 14:19:13.133729 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:19:13.133683    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 20 14:19:23.134505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:19:23.134460    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 20 14:19:33.134494 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:19:33.134446    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 20 14:19:43.133523 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:19:43.133479    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 20 14:19:53.134614 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:19:53.134541    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 20 14:20:03.133585 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:03.133534    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 20 14:20:05.656739 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:05.656703    2550 generic.go:358] "Generic (PLEG): container finished" podID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerID="18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06" exitCode=0
May 20 14:20:05.657171 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:05.656777    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" event={"ID":"45d338f4-9c17-4a22-b0a6-e18884fbf43d","Type":"ContainerDied","Data":"18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06"}
May 20 14:20:06.662490 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:06.662451    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" event={"ID":"45d338f4-9c17-4a22-b0a6-e18884fbf43d","Type":"ContainerStarted","Data":"20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd"}
May 20 14:20:06.690164 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:06.690101    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podStartSLOduration=124.690078858 podStartE2EDuration="2m4.690078858s" podCreationTimestamp="2026-05-20 14:18:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:20:06.68702527 +0000 UTC m=+1784.813408424" watchObservedRunningTime="2026-05-20 14:20:06.690078858 +0000 UTC m=+1784.816462013"
May 20 14:20:13.117718 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:13.117612    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:20:13.117718 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:13.117675    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:20:13.118233 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:13.117997    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
May 20 14:20:13.134164 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:13.134115    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 20 14:20:23.117976 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:23.117923    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
May 20 14:20:23.134160 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:23.134115    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 20 14:20:33.117613 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:33.117555    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
May 20 14:20:33.133845 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:33.133802    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
May 20 14:20:43.117480 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:43.117429    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
May 20 14:20:43.144034 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:43.144001    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:20:43.151859 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:43.151822    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:20:53.117511 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:20:53.117459    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
May 20 14:21:03.117538 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:21:03.117480    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
May 20 14:21:13.117634 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:21:13.117571    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
May 20 14:21:23.117301 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:21:23.117254    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
May 20 14:21:33.117183 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:21:33.117134    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8001/health\": dial tcp 10.133.0.42:8001: connect: connection refused"
May 20 14:21:43.126611 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:21:43.126528    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:21:43.140126 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:21:43.140097    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:21:51.098109 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:21:51.098063    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"]
May 20 14:21:51.098692 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:21:51.098546    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main" containerID="cri-o://20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd" gracePeriod=30
May 20 14:21:51.108832 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:21:51.108803    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"]
May 20 14:21:51.109198 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:21:51.109144    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main" containerID="cri-o://0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464" gracePeriod=30
May 20 14:22:04.586712 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.586677    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"]
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587094    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="llm-d-routing-sidecar"
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587108    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="llm-d-routing-sidecar"
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587121    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main"
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587128    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main"
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587141    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main"
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587150    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main"
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587168    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="storage-initializer"
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587174    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="storage-initializer"
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587182    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="storage-initializer"
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587188    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="storage-initializer"
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587268    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="main"
May 20 14:22:04.587275 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587280    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="2e665f73-e48f-481c-a32c-a89fd5732140" containerName="main"
May 20 14:22:04.587849 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.587290    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="8d72dfee-d8b4-4fa3-8fc9-66673d88f76c" containerName="llm-d-routing-sidecar"
May 20 14:22:04.591014 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.590993    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.593928 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.593900    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-pr8dz\""
May 20 14:22:04.594018 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.593911    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 20 14:22:04.601746 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.601721    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"]
May 20 14:22:04.605500 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.605478    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.605668 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.605648    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"]
May 20 14:22:04.620283 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.620257    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"]
May 20 14:22:04.630431 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.625961    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8982c4e0-0df7-44cb-9787-a7f65f289e46-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.630431 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.626057    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.630431 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.626088    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.630431 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.626123    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvpc\" (UniqueName: \"kubernetes.io/projected/8982c4e0-0df7-44cb-9787-a7f65f289e46-kube-api-access-8fvpc\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.630431 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.626505    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.630431 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.626582    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.630431 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.626644    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-home\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.727785 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.727741    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8982c4e0-0df7-44cb-9787-a7f65f289e46-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.727785 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.727790    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-home\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.728056 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.727831    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.728056 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.727858    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.728056 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.727889    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-model-cache\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.728056 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728004    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxd6t\" (UniqueName: \"kubernetes.io/projected/58817abd-ef34-43f6-84dc-f2a36b788fc0-kube-api-access-dxd6t\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.728056 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728053    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58817abd-ef34-43f6-84dc-f2a36b788fc0-tls-certs\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.728313 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728107    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.728313 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728130    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvpc\" (UniqueName: \"kubernetes.io/projected/8982c4e0-0df7-44cb-9787-a7f65f289e46-kube-api-access-8fvpc\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.728313 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728164    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.728313 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728196    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.728313 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728244    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-home\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.728313 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728276    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-dshm\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.728313 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728286    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.728782 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728326    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.728782 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728331    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.728782 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728614    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-home\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.728913 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.728825    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.730235 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.730212    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.730454 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.730435    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8982c4e0-0df7-44cb-9787-a7f65f289e46-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.736170 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.736144    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvpc\" (UniqueName: \"kubernetes.io/projected/8982c4e0-0df7-44cb-9787-a7f65f289e46-kube-api-access-8fvpc\") pod \"router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:04.829390 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.829326    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.829544 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.829409    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-dshm\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.829544 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.829428    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.829544 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.829453    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-home\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.829544 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.829489    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-model-cache\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.829544 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.829521    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dxd6t\" (UniqueName: \"kubernetes.io/projected/58817abd-ef34-43f6-84dc-f2a36b788fc0-kube-api-access-dxd6t\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.829812 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.829558    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58817abd-ef34-43f6-84dc-f2a36b788fc0-tls-certs\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.829812 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.829784    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.829916 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.829848    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-tmp-dir\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.830005 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.829977    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-home\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.830005 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.829991    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-model-cache\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.831648 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.831629    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-dshm\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.832081 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.832061    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58817abd-ef34-43f6-84dc-f2a36b788fc0-tls-certs\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.838062 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.837985    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxd6t\" (UniqueName: \"kubernetes.io/projected/58817abd-ef34-43f6-84dc-f2a36b788fc0-kube-api-access-dxd6t\") pod \"router-with-refs-pd-test-kserve-8c9cd8f-v6xzj\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.902438 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.902399    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:04.916419 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:04.916386    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:05.051577 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:05.051533    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"]
May 20 14:22:05.053229 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:22:05.053197    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58817abd_ef34_43f6_84dc_f2a36b788fc0.slice/crio-21d5365dcbd03ecff5c38549fac841ed18592e0f62da295e518980097e9794ca WatchSource:0}: Error finding container 21d5365dcbd03ecff5c38549fac841ed18592e0f62da295e518980097e9794ca: Status 404 returned error can't find the container with id 21d5365dcbd03ecff5c38549fac841ed18592e0f62da295e518980097e9794ca
May 20 14:22:05.077233 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:05.077203    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"]
May 20 14:22:05.094756 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:22:05.094723    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8982c4e0_0df7_44cb_9787_a7f65f289e46.slice/crio-432b648c8d61eb650b9f119e55a9499d8f2eefe28087dba5628734a0f66efe44 WatchSource:0}: Error finding container 432b648c8d61eb650b9f119e55a9499d8f2eefe28087dba5628734a0f66efe44: Status 404 returned error can't find the container with id 432b648c8d61eb650b9f119e55a9499d8f2eefe28087dba5628734a0f66efe44
May 20 14:22:05.115792 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:05.115759    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" event={"ID":"8982c4e0-0df7-44cb-9787-a7f65f289e46","Type":"ContainerStarted","Data":"432b648c8d61eb650b9f119e55a9499d8f2eefe28087dba5628734a0f66efe44"}
May 20 14:22:05.117297 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:05.117270    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" event={"ID":"58817abd-ef34-43f6-84dc-f2a36b788fc0","Type":"ContainerStarted","Data":"21d5365dcbd03ecff5c38549fac841ed18592e0f62da295e518980097e9794ca"}
May 20 14:22:06.124129 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:06.124090    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" event={"ID":"8982c4e0-0df7-44cb-9787-a7f65f289e46","Type":"ContainerStarted","Data":"df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524"}
May 20 14:22:06.125639 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:06.125609    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" event={"ID":"58817abd-ef34-43f6-84dc-f2a36b788fc0","Type":"ContainerStarted","Data":"d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586"}
May 20 14:22:06.125778 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:06.125714    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:07.131425 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:07.131380    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" event={"ID":"58817abd-ef34-43f6-84dc-f2a36b788fc0","Type":"ContainerStarted","Data":"d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b"}
May 20 14:22:10.143780 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:10.143747    2550 generic.go:358] "Generic (PLEG): container finished" podID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerID="df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524" exitCode=0
May 20 14:22:10.144781 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:10.143824    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" event={"ID":"8982c4e0-0df7-44cb-9787-a7f65f289e46","Type":"ContainerDied","Data":"df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524"}
May 20 14:22:11.149003 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:11.148967    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" event={"ID":"8982c4e0-0df7-44cb-9787-a7f65f289e46","Type":"ContainerStarted","Data":"d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971"}
May 20 14:22:11.150696 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:11.150669    2550 generic.go:358] "Generic (PLEG): container finished" podID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerID="d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b" exitCode=0
May 20 14:22:11.150796 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:11.150730    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" event={"ID":"58817abd-ef34-43f6-84dc-f2a36b788fc0","Type":"ContainerDied","Data":"d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b"}
May 20 14:22:11.174738 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:11.174676    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podStartSLOduration=7.1746575 podStartE2EDuration="7.1746575s" podCreationTimestamp="2026-05-20 14:22:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:22:11.171711749 +0000 UTC m=+1909.298094895" watchObservedRunningTime="2026-05-20 14:22:11.1746575 +0000 UTC m=+1909.301040654"
May 20 14:22:12.156825 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:12.156784    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" event={"ID":"58817abd-ef34-43f6-84dc-f2a36b788fc0","Type":"ContainerStarted","Data":"b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33"}
May 20 14:22:12.180464 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:12.180406    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podStartSLOduration=8.180387271 podStartE2EDuration="8.180387271s" podCreationTimestamp="2026-05-20 14:22:04 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:22:12.177883155 +0000 UTC m=+1910.304266308" watchObservedRunningTime="2026-05-20 14:22:12.180387271 +0000 UTC m=+1910.306770424"
May 20 14:22:14.903220 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:14.903179    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:14.903703 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:14.903239    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:14.904695 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:14.904658    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:22:14.917002 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:14.916972    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:14.917158 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:14.917020    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:22:14.918605 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:14.918569    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:22:21.099453 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.099378    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="llm-d-routing-sidecar" containerID="cri-o://5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef" gracePeriod=2
May 20 14:22:21.433973 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.433949    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:22:21.437421 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.437397    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46_45d338f4-9c17-4a22-b0a6-e18884fbf43d/main/0.log"
May 20 14:22:21.438151 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.438130    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:22:21.505576 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.505547    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mfgl\" (UniqueName: \"kubernetes.io/projected/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kube-api-access-8mfgl\") pod \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") "
May 20 14:22:21.505576 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.505584    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kserve-provision-location\") pod \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") "
May 20 14:22:21.505829 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.505630    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc900de-4093-4423-b190-b48c648a849e-tls-certs\") pod \"ebc900de-4093-4423-b190-b48c648a849e\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") "
May 20 14:22:21.505829 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.505650    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-dshm\") pod \"ebc900de-4093-4423-b190-b48c648a849e\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") "
May 20 14:22:21.505829 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.505810    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-kserve-provision-location\") pod \"ebc900de-4093-4423-b190-b48c648a849e\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") "
May 20 14:22:21.506004 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.505850    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kc4jd\" (UniqueName: \"kubernetes.io/projected/ebc900de-4093-4423-b190-b48c648a849e-kube-api-access-kc4jd\") pod \"ebc900de-4093-4423-b190-b48c648a849e\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") "
May 20 14:22:21.506004 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.505886    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tmp-dir\") pod \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") "
May 20 14:22:21.506004 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.505918    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-dshm\") pod \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") "
May 20 14:22:21.506004 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.505944    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-model-cache\") pod \"ebc900de-4093-4423-b190-b48c648a849e\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") "
May 20 14:22:21.506004 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.505966    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-home\") pod \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") "
May 20 14:22:21.506247 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.506074    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-home\") pod \"ebc900de-4093-4423-b190-b48c648a849e\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") "
May 20 14:22:21.506247 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.506117    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-tmp-dir\") pod \"ebc900de-4093-4423-b190-b48c648a849e\" (UID: \"ebc900de-4093-4423-b190-b48c648a849e\") "
May 20 14:22:21.506247 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.506145    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tls-certs\") pod \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") "
May 20 14:22:21.506247 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.506190    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-model-cache\") pod \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\" (UID: \"45d338f4-9c17-4a22-b0a6-e18884fbf43d\") "
May 20 14:22:21.507957 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.507921    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-model-cache" (OuterVolumeSpecName: "model-cache") pod "45d338f4-9c17-4a22-b0a6-e18884fbf43d" (UID: "45d338f4-9c17-4a22-b0a6-e18884fbf43d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:22:21.508298 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.508265    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc900de-4093-4423-b190-b48c648a849e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebc900de-4093-4423-b190-b48c648a849e" (UID: "ebc900de-4093-4423-b190-b48c648a849e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:22:21.508882 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.508712    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-dshm" (OuterVolumeSpecName: "dshm") pod "ebc900de-4093-4423-b190-b48c648a849e" (UID: "ebc900de-4093-4423-b190-b48c648a849e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:22:21.509173 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.508980    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-home" (OuterVolumeSpecName: "home") pod "ebc900de-4093-4423-b190-b48c648a849e" (UID: "ebc900de-4093-4423-b190-b48c648a849e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:22:21.509575 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.509321    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebc900de-4093-4423-b190-b48c648a849e" (UID: "ebc900de-4093-4423-b190-b48c648a849e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:22:21.510637 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.509952    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-home" (OuterVolumeSpecName: "home") pod "45d338f4-9c17-4a22-b0a6-e18884fbf43d" (UID: "45d338f4-9c17-4a22-b0a6-e18884fbf43d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:22:21.510740 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.510679    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-dshm" (OuterVolumeSpecName: "dshm") pod "45d338f4-9c17-4a22-b0a6-e18884fbf43d" (UID: "45d338f4-9c17-4a22-b0a6-e18884fbf43d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:22:21.511455 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.511419    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kube-api-access-8mfgl" (OuterVolumeSpecName: "kube-api-access-8mfgl") pod "45d338f4-9c17-4a22-b0a6-e18884fbf43d" (UID: "45d338f4-9c17-4a22-b0a6-e18884fbf43d"). InnerVolumeSpecName "kube-api-access-8mfgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:22:21.512261 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.512231    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc900de-4093-4423-b190-b48c648a849e-kube-api-access-kc4jd" (OuterVolumeSpecName: "kube-api-access-kc4jd") pod "ebc900de-4093-4423-b190-b48c648a849e" (UID: "ebc900de-4093-4423-b190-b48c648a849e"). InnerVolumeSpecName "kube-api-access-kc4jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:22:21.513297 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.513272    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45d338f4-9c17-4a22-b0a6-e18884fbf43d" (UID: "45d338f4-9c17-4a22-b0a6-e18884fbf43d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:22:21.528090 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.528047    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45d338f4-9c17-4a22-b0a6-e18884fbf43d" (UID: "45d338f4-9c17-4a22-b0a6-e18884fbf43d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:22:21.531937 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.531895    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebc900de-4093-4423-b190-b48c648a849e" (UID: "ebc900de-4093-4423-b190-b48c648a849e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:22:21.568217 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.568169    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45d338f4-9c17-4a22-b0a6-e18884fbf43d" (UID: "45d338f4-9c17-4a22-b0a6-e18884fbf43d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:22:21.572440 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.572373    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebc900de-4093-4423-b190-b48c648a849e" (UID: "ebc900de-4093-4423-b190-b48c648a849e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:22:21.607921 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.607870    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mfgl\" (UniqueName: \"kubernetes.io/projected/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kube-api-access-8mfgl\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.607921 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.607904    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.607921 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.607919    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebc900de-4093-4423-b190-b48c648a849e-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.608246 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.607933    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.608246 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.607946    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.608246 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.607959    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc4jd\" (UniqueName: \"kubernetes.io/projected/ebc900de-4093-4423-b190-b48c648a849e-kube-api-access-kc4jd\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.608246 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.607972    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.608246 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.607986    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.608246 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.607998    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.608246 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.608008    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.608246 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.608019    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.608246 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.608030    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc900de-4093-4423-b190-b48c648a849e-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.608246 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.608042    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d338f4-9c17-4a22-b0a6-e18884fbf43d-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:21.608246 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:21.608054    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d338f4-9c17-4a22-b0a6-e18884fbf43d-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:22:22.199670 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.199631    2550 generic.go:358] "Generic (PLEG): container finished" podID="ebc900de-4093-4423-b190-b48c648a849e" containerID="0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464" exitCode=137
May 20 14:22:22.200215 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.199710    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"
May 20 14:22:22.200215 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.199703    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" event={"ID":"ebc900de-4093-4423-b190-b48c648a849e","Type":"ContainerDied","Data":"0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464"}
May 20 14:22:22.200215 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.199871    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj" event={"ID":"ebc900de-4093-4423-b190-b48c648a849e","Type":"ContainerDied","Data":"a96bf39060966a9789572f375e657c1f3557228287a70d216d64dcb7c2cf033f"}
May 20 14:22:22.200215 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.199898    2550 scope.go:117] "RemoveContainer" containerID="0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464"
May 20 14:22:22.201439 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.201035    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46_45d338f4-9c17-4a22-b0a6-e18884fbf43d/main/0.log"
May 20 14:22:22.201759 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.201732    2550 generic.go:358] "Generic (PLEG): container finished" podID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerID="20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd" exitCode=137
May 20 14:22:22.201759 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.201757    2550 generic.go:358] "Generic (PLEG): container finished" podID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerID="5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef" exitCode=0
May 20 14:22:22.201949 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.201819    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" event={"ID":"45d338f4-9c17-4a22-b0a6-e18884fbf43d","Type":"ContainerDied","Data":"20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd"}
May 20 14:22:22.201949 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.201832    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"
May 20 14:22:22.201949 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.201845    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" event={"ID":"45d338f4-9c17-4a22-b0a6-e18884fbf43d","Type":"ContainerDied","Data":"5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef"}
May 20 14:22:22.201949 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.201876    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46" event={"ID":"45d338f4-9c17-4a22-b0a6-e18884fbf43d","Type":"ContainerDied","Data":"82841eb79d4a9386d88f887fd2e44342d1f3168807a9213badab3f2ce42e9287"}
May 20 14:22:22.209511 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.209304    2550 scope.go:117] "RemoveContainer" containerID="e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa"
May 20 14:22:22.224478 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.224447    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"]
May 20 14:22:22.228267 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.228242    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-75d85d5cb8-crdgj"]
May 20 14:22:22.228668 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.228650    2550 scope.go:117] "RemoveContainer" containerID="0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464"
May 20 14:22:22.229008 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:22:22.228977    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464\": container with ID starting with 0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464 not found: ID does not exist" containerID="0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464"
May 20 14:22:22.229095 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.229013    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464"} err="failed to get container status \"0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464\": rpc error: code = NotFound desc = could not find container \"0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464\": container with ID starting with 0efc597030e0f93989e1df51f6460ed475c0c221f36d09dfb0dfcdde8c0d1464 not found: ID does not exist"
May 20 14:22:22.229095 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.229038    2550 scope.go:117] "RemoveContainer" containerID="e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa"
May 20 14:22:22.229349 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:22:22.229305    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa\": container with ID starting with e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa not found: ID does not exist" containerID="e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa"
May 20 14:22:22.229421 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.229361    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa"} err="failed to get container status \"e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa\": rpc error: code = NotFound desc = could not find container \"e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa\": container with ID starting with e79f157896198bc85df8b82e7216d05b07cfd3aed6deb6b90a3535158f6233fa not found: ID does not exist"
May 20 14:22:22.229421 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.229386    2550 scope.go:117] "RemoveContainer" containerID="20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd"
May 20 14:22:22.238862 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.238257    2550 scope.go:117] "RemoveContainer" containerID="18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06"
May 20 14:22:22.241442 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.241414    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"]
May 20 14:22:22.249901 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.249869    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-c86f76cb9-dcj46"]
May 20 14:22:22.292239 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.292211    2550 scope.go:117] "RemoveContainer" containerID="5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef"
May 20 14:22:22.301031 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.301000    2550 scope.go:117] "RemoveContainer" containerID="20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd"
May 20 14:22:22.301384 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:22:22.301355    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd\": container with ID starting with 20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd not found: ID does not exist" containerID="20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd"
May 20 14:22:22.301474 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.301397    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd"} err="failed to get container status \"20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd\": rpc error: code = NotFound desc = could not find container \"20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd\": container with ID starting with 20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd not found: ID does not exist"
May 20 14:22:22.301474 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.301426    2550 scope.go:117] "RemoveContainer" containerID="18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06"
May 20 14:22:22.301744 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:22:22.301724    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06\": container with ID starting with 18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06 not found: ID does not exist" containerID="18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06"
May 20 14:22:22.301802 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.301754    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06"} err="failed to get container status \"18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06\": rpc error: code = NotFound desc = could not find container \"18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06\": container with ID starting with 18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06 not found: ID does not exist"
May 20 14:22:22.301802 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.301774    2550 scope.go:117] "RemoveContainer" containerID="5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef"
May 20 14:22:22.302037 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:22:22.302020    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef\": container with ID starting with 5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef not found: ID does not exist" containerID="5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef"
May 20 14:22:22.302076 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.302040    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef"} err="failed to get container status \"5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef\": rpc error: code = NotFound desc = could not find container \"5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef\": container with ID starting with 5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef not found: ID does not exist"
May 20 14:22:22.302076 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.302052    2550 scope.go:117] "RemoveContainer" containerID="20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd"
May 20 14:22:22.302286 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.302270    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd"} err="failed to get container status \"20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd\": rpc error: code = NotFound desc = could not find container \"20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd\": container with ID starting with 20d7fa0fc874ee5807383e14fa46daa64a6a29b23fc2666621bc1b550a3936fd not found: ID does not exist"
May 20 14:22:22.302364 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.302286    2550 scope.go:117] "RemoveContainer" containerID="18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06"
May 20 14:22:22.302616 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.302586    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06"} err="failed to get container status \"18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06\": rpc error: code = NotFound desc = could not find container \"18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06\": container with ID starting with 18089786054f97caadbecdb1d74aae3ad51f62fb9a758e3d4c8e28a80a76af06 not found: ID does not exist"
May 20 14:22:22.302698 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.302618    2550 scope.go:117] "RemoveContainer" containerID="5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef"
May 20 14:22:22.302894 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.302858    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef"} err="failed to get container status \"5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef\": rpc error: code = NotFound desc = could not find container \"5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef\": container with ID starting with 5d4b68cb34740905261d6996f34420d39744c6d5eb09dcb7bab9713e39f02aef not found: ID does not exist"
May 20 14:22:22.469195 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.469104    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" path="/var/lib/kubelet/pods/45d338f4-9c17-4a22-b0a6-e18884fbf43d/volumes"
May 20 14:22:22.469712 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:22.469690    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc900de-4093-4423-b190-b48c648a849e" path="/var/lib/kubelet/pods/ebc900de-4093-4423-b190-b48c648a849e/volumes"
May 20 14:22:24.903554 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:24.903503    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:22:24.916746 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:24.916716    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:22:24.916911 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:24.916843    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:22:34.903757 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:34.903707    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:22:34.917411 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:34.917358    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:22:44.903789 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:44.903739    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:22:44.917570 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:44.917522    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:22:54.903143 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:54.903087    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:22:54.917388 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:22:54.917330    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:23:04.903083 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:04.903024    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:23:04.917114 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:04.917075    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:23:14.903266 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:14.903158    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:23:14.916972 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:14.916930    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:23:24.903116 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:24.903068    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:23:24.917304 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:24.917258    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:23:34.903816 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:34.903737    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:23:34.916904 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:34.916869    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:23:44.903804 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:44.903757    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:23:44.918163 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:44.918113    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:23:54.903014 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:54.902959    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:23:54.917606 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:23:54.917563    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:24:04.903292 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:04.903235    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:24:04.917517 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:04.917480    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:24:14.903352 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:14.903299    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:24:14.917765 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:14.917726    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:24:24.903293 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:24.903236    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8001/health\": dial tcp 10.133.0.44:8001: connect: connection refused"
May 20 14:24:24.918215 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:24.918170    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
May 20 14:24:34.913082 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:34.913048    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:24:34.926199 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:34.926174    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:24:34.931801 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:34.931780    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:24:34.934415 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:34.934395    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:24:47.187203 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:47.187162    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"]
May 20 14:24:47.188127 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:47.188070    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main" containerID="cri-o://b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33" gracePeriod=30
May 20 14:24:47.198353 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:47.198314    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"]
May 20 14:24:47.198704 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:47.198671    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main" containerID="cri-o://d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971" gracePeriod=30
May 20 14:24:50.852409 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852375    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"]
May 20 14:24:50.852809 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852746    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="storage-initializer"
May 20 14:24:50.852809 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852762    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="storage-initializer"
May 20 14:24:50.852809 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852785    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="llm-d-routing-sidecar"
May 20 14:24:50.852809 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852794    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="llm-d-routing-sidecar"
May 20 14:24:50.852954 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852823    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main"
May 20 14:24:50.852954 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852830    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main"
May 20 14:24:50.852954 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852841    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="storage-initializer"
May 20 14:24:50.852954 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852846    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="storage-initializer"
May 20 14:24:50.852954 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852856    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main"
May 20 14:24:50.852954 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852861    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main"
May 20 14:24:50.852954 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852943    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="llm-d-routing-sidecar"
May 20 14:24:50.852954 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852952    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="ebc900de-4093-4423-b190-b48c648a849e" containerName="main"
May 20 14:24:50.853189 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.852959    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="45d338f4-9c17-4a22-b0a6-e18884fbf43d" containerName="main"
May 20 14:24:50.857608 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.857587    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:50.860114 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.860089    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
May 20 14:24:50.868874 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.868847    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"]
May 20 14:24:50.948576 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.948537    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68pct\" (UniqueName: \"kubernetes.io/projected/7e8bd313-d746-4635-86c1-1c31adcf806d-kube-api-access-68pct\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:50.948576 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.948579    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e8bd313-d746-4635-86c1-1c31adcf806d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:50.948800 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.948682    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:50.948800 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.948722    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:50.948800 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.948748    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:50.948922 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.948807    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:50.948922 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:50.948836    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.050084 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.050043    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.050255 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.050105    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-68pct\" (UniqueName: \"kubernetes.io/projected/7e8bd313-d746-4635-86c1-1c31adcf806d-kube-api-access-68pct\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.050255 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.050136    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e8bd313-d746-4635-86c1-1c31adcf806d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.050255 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.050195    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.050255 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.050229    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.050435 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.050265    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.050435 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.050304    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.050688 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.050665    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.050777 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.050734    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.050777 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.050768    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.050902 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.050861    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.052670 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.052641    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.053014 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.052992    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e8bd313-d746-4635-86c1-1c31adcf806d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.058976 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.058947    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68pct\" (UniqueName: \"kubernetes.io/projected/7e8bd313-d746-4635-86c1-1c31adcf806d-kube-api-access-68pct\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.170461 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.170329    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:24:51.311203 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.310586    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"]
May 20 14:24:51.314632 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:24:51.314596    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8bd313_d746_4635_86c1_1c31adcf806d.slice/crio-941d49a614f11713420b591498f01b03a13c01fcf1cb31fbf0766ee969e824b6 WatchSource:0}: Error finding container 941d49a614f11713420b591498f01b03a13c01fcf1cb31fbf0766ee969e824b6: Status 404 returned error can't find the container with id 941d49a614f11713420b591498f01b03a13c01fcf1cb31fbf0766ee969e824b6
May 20 14:24:51.317390 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.317367    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 14:24:51.760216 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.760170    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" event={"ID":"7e8bd313-d746-4635-86c1-1c31adcf806d","Type":"ContainerStarted","Data":"975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d"}
May 20 14:24:51.760406 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:51.760222    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" event={"ID":"7e8bd313-d746-4635-86c1-1c31adcf806d","Type":"ContainerStarted","Data":"941d49a614f11713420b591498f01b03a13c01fcf1cb31fbf0766ee969e824b6"}
May 20 14:24:56.778812 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:56.778781    2550 generic.go:358] "Generic (PLEG): container finished" podID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerID="975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d" exitCode=0
May 20 14:24:56.779174 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:56.778857    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" event={"ID":"7e8bd313-d746-4635-86c1-1c31adcf806d","Type":"ContainerDied","Data":"975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d"}
May 20 14:24:57.784229 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:57.784191    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" event={"ID":"7e8bd313-d746-4635-86c1-1c31adcf806d","Type":"ContainerStarted","Data":"828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc"}
May 20 14:24:57.804782 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:24:57.804713    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podStartSLOduration=7.804694289 podStartE2EDuration="7.804694289s" podCreationTimestamp="2026-05-20 14:24:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:24:57.803394582 +0000 UTC m=+2075.929777737" watchObservedRunningTime="2026-05-20 14:24:57.804694289 +0000 UTC m=+2075.931077443"
May 20 14:25:01.170987 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:01.170939    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:25:01.170987 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:01.170996    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:25:01.172744 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:01.172707    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
May 20 14:25:11.171100 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:11.171057    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
May 20 14:25:17.188661 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.188598    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="llm-d-routing-sidecar" containerID="cri-o://d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586" gracePeriod=2
May 20 14:25:17.492312 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.492281    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:25:17.495996 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.495971    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-8c9cd8f-v6xzj_58817abd-ef34-43f6-84dc-f2a36b788fc0/main/0.log"
May 20 14:25:17.496765 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.496745    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:25:17.614094 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614059    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58817abd-ef34-43f6-84dc-f2a36b788fc0-tls-certs\") pod \"58817abd-ef34-43f6-84dc-f2a36b788fc0\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") "
May 20 14:25:17.614094 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614097    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-dshm\") pod \"58817abd-ef34-43f6-84dc-f2a36b788fc0\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") "
May 20 14:25:17.614360 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614122    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-tmp-dir\") pod \"8982c4e0-0df7-44cb-9787-a7f65f289e46\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") "
May 20 14:25:17.614360 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614138    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-kserve-provision-location\") pod \"58817abd-ef34-43f6-84dc-f2a36b788fc0\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") "
May 20 14:25:17.614360 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614176    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fvpc\" (UniqueName: \"kubernetes.io/projected/8982c4e0-0df7-44cb-9787-a7f65f289e46-kube-api-access-8fvpc\") pod \"8982c4e0-0df7-44cb-9787-a7f65f289e46\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") "
May 20 14:25:17.614360 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614192    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-tmp-dir\") pod \"58817abd-ef34-43f6-84dc-f2a36b788fc0\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") "
May 20 14:25:17.614360 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614234    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-kserve-provision-location\") pod \"8982c4e0-0df7-44cb-9787-a7f65f289e46\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") "
May 20 14:25:17.614360 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614254    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-home\") pod \"58817abd-ef34-43f6-84dc-f2a36b788fc0\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") "
May 20 14:25:17.614360 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614304    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-model-cache\") pod \"58817abd-ef34-43f6-84dc-f2a36b788fc0\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") "
May 20 14:25:17.614733 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614364    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8982c4e0-0df7-44cb-9787-a7f65f289e46-tls-certs\") pod \"8982c4e0-0df7-44cb-9787-a7f65f289e46\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") "
May 20 14:25:17.614733 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614389    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-dshm\") pod \"8982c4e0-0df7-44cb-9787-a7f65f289e46\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") "
May 20 14:25:17.614733 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614440    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-home\") pod \"8982c4e0-0df7-44cb-9787-a7f65f289e46\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") "
May 20 14:25:17.614733 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614474    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxd6t\" (UniqueName: \"kubernetes.io/projected/58817abd-ef34-43f6-84dc-f2a36b788fc0-kube-api-access-dxd6t\") pod \"58817abd-ef34-43f6-84dc-f2a36b788fc0\" (UID: \"58817abd-ef34-43f6-84dc-f2a36b788fc0\") "
May 20 14:25:17.614733 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614512    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-model-cache\") pod \"8982c4e0-0df7-44cb-9787-a7f65f289e46\" (UID: \"8982c4e0-0df7-44cb-9787-a7f65f289e46\") "
May 20 14:25:17.614733 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614659    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "58817abd-ef34-43f6-84dc-f2a36b788fc0" (UID: "58817abd-ef34-43f6-84dc-f2a36b788fc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:17.615042 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614859    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.615042 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.614862    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-model-cache" (OuterVolumeSpecName: "model-cache") pod "8982c4e0-0df7-44cb-9787-a7f65f289e46" (UID: "8982c4e0-0df7-44cb-9787-a7f65f289e46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:17.616104 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.615824    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-home" (OuterVolumeSpecName: "home") pod "8982c4e0-0df7-44cb-9787-a7f65f289e46" (UID: "8982c4e0-0df7-44cb-9787-a7f65f289e46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:17.617286 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.617260    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-dshm" (OuterVolumeSpecName: "dshm") pod "58817abd-ef34-43f6-84dc-f2a36b788fc0" (UID: "58817abd-ef34-43f6-84dc-f2a36b788fc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:17.617751 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.617555    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58817abd-ef34-43f6-84dc-f2a36b788fc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58817abd-ef34-43f6-84dc-f2a36b788fc0" (UID: "58817abd-ef34-43f6-84dc-f2a36b788fc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:25:17.619042 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.619009    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8982c4e0-0df7-44cb-9787-a7f65f289e46-kube-api-access-8fvpc" (OuterVolumeSpecName: "kube-api-access-8fvpc") pod "8982c4e0-0df7-44cb-9787-a7f65f289e46" (UID: "8982c4e0-0df7-44cb-9787-a7f65f289e46"). InnerVolumeSpecName "kube-api-access-8fvpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:25:17.619151 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.619055    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-dshm" (OuterVolumeSpecName: "dshm") pod "8982c4e0-0df7-44cb-9787-a7f65f289e46" (UID: "8982c4e0-0df7-44cb-9787-a7f65f289e46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:17.619634 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.619610    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-home" (OuterVolumeSpecName: "home") pod "58817abd-ef34-43f6-84dc-f2a36b788fc0" (UID: "58817abd-ef34-43f6-84dc-f2a36b788fc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:17.619850 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.619824    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58817abd-ef34-43f6-84dc-f2a36b788fc0-kube-api-access-dxd6t" (OuterVolumeSpecName: "kube-api-access-dxd6t") pod "58817abd-ef34-43f6-84dc-f2a36b788fc0" (UID: "58817abd-ef34-43f6-84dc-f2a36b788fc0"). InnerVolumeSpecName "kube-api-access-dxd6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:25:17.620097 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.620079    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8982c4e0-0df7-44cb-9787-a7f65f289e46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8982c4e0-0df7-44cb-9787-a7f65f289e46" (UID: "8982c4e0-0df7-44cb-9787-a7f65f289e46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:25:17.629723 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.629671    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8982c4e0-0df7-44cb-9787-a7f65f289e46" (UID: "8982c4e0-0df7-44cb-9787-a7f65f289e46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:17.630779 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.630757    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58817abd-ef34-43f6-84dc-f2a36b788fc0" (UID: "58817abd-ef34-43f6-84dc-f2a36b788fc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:17.686717 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.686649    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58817abd-ef34-43f6-84dc-f2a36b788fc0" (UID: "58817abd-ef34-43f6-84dc-f2a36b788fc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:17.689251 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.689221    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8982c4e0-0df7-44cb-9787-a7f65f289e46" (UID: "8982c4e0-0df7-44cb-9787-a7f65f289e46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:25:17.715972 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.715940    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.715972 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.715971    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.716121 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.715982    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8982c4e0-0df7-44cb-9787-a7f65f289e46-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.716121 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.715992    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.716121 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.716001    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.716121 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.716010    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxd6t\" (UniqueName: \"kubernetes.io/projected/58817abd-ef34-43f6-84dc-f2a36b788fc0-kube-api-access-dxd6t\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.716121 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.716017    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.716121 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.716028    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58817abd-ef34-43f6-84dc-f2a36b788fc0-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.716121 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.716035    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.716121 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.716043    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8982c4e0-0df7-44cb-9787-a7f65f289e46-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.716121 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.716051    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.716121 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.716060    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fvpc\" (UniqueName: \"kubernetes.io/projected/8982c4e0-0df7-44cb-9787-a7f65f289e46-kube-api-access-8fvpc\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.716121 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.716068    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58817abd-ef34-43f6-84dc-f2a36b788fc0-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:25:17.859080 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.859044    2550 generic.go:358] "Generic (PLEG): container finished" podID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerID="d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971" exitCode=137
May 20 14:25:17.859273 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.859130    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" event={"ID":"8982c4e0-0df7-44cb-9787-a7f65f289e46","Type":"ContainerDied","Data":"d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971"}
May 20 14:25:17.859273 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.859167    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"
May 20 14:25:17.859273 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.859177    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp" event={"ID":"8982c4e0-0df7-44cb-9787-a7f65f289e46","Type":"ContainerDied","Data":"432b648c8d61eb650b9f119e55a9499d8f2eefe28087dba5628734a0f66efe44"}
May 20 14:25:17.859273 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.859205    2550 scope.go:117] "RemoveContainer" containerID="d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971"
May 20 14:25:17.860488 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.860472    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-8c9cd8f-v6xzj_58817abd-ef34-43f6-84dc-f2a36b788fc0/main/0.log"
May 20 14:25:17.861111 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.861092    2550 generic.go:358] "Generic (PLEG): container finished" podID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerID="b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33" exitCode=137
May 20 14:25:17.861111 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.861108    2550 generic.go:358] "Generic (PLEG): container finished" podID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerID="d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586" exitCode=0
May 20 14:25:17.861276 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.861164    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" event={"ID":"58817abd-ef34-43f6-84dc-f2a36b788fc0","Type":"ContainerDied","Data":"b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33"}
May 20 14:25:17.861276 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.861177    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"
May 20 14:25:17.861276 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.861184    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" event={"ID":"58817abd-ef34-43f6-84dc-f2a36b788fc0","Type":"ContainerDied","Data":"d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586"}
May 20 14:25:17.861276 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.861198    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj" event={"ID":"58817abd-ef34-43f6-84dc-f2a36b788fc0","Type":"ContainerDied","Data":"21d5365dcbd03ecff5c38549fac841ed18592e0f62da295e518980097e9794ca"}
May 20 14:25:17.868424 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.868397    2550 scope.go:117] "RemoveContainer" containerID="df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524"
May 20 14:25:17.889592 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.889552    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"]
May 20 14:25:17.892861 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.892823    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-8c9cd8f-v6xzj"]
May 20 14:25:17.893661 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.893645    2550 scope.go:117] "RemoveContainer" containerID="d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971"
May 20 14:25:17.893961 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:25:17.893942    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971\": container with ID starting with d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971 not found: ID does not exist" containerID="d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971"
May 20 14:25:17.894056 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.893966    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971"} err="failed to get container status \"d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971\": rpc error: code = NotFound desc = could not find container \"d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971\": container with ID starting with d02416ae88a2e3a94afb4ac1c1e46253c6f13c48891d62c78377a5a4b7656971 not found: ID does not exist"
May 20 14:25:17.894056 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.893985    2550 scope.go:117] "RemoveContainer" containerID="df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524"
May 20 14:25:17.894266 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:25:17.894247    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524\": container with ID starting with df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524 not found: ID does not exist" containerID="df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524"
May 20 14:25:17.894308 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.894272    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524"} err="failed to get container status \"df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524\": rpc error: code = NotFound desc = could not find container \"df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524\": container with ID starting with df5ab8f2d732e1376a8089e91354801c6d57083e0e649dda435b47b8c3167524 not found: ID does not exist"
May 20 14:25:17.894308 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.894289    2550 scope.go:117] "RemoveContainer" containerID="b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33"
May 20 14:25:17.903621 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.903601    2550 scope.go:117] "RemoveContainer" containerID="d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b"
May 20 14:25:17.904379 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.904317    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"]
May 20 14:25:17.908613 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.908587    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-8597c688bd-wbhgp"]
May 20 14:25:17.969416 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.969391    2550 scope.go:117] "RemoveContainer" containerID="d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586"
May 20 14:25:17.978239 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.978215    2550 scope.go:117] "RemoveContainer" containerID="b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33"
May 20 14:25:17.978620 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:25:17.978598    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33\": container with ID starting with b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33 not found: ID does not exist" containerID="b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33"
May 20 14:25:17.978698 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.978630    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33"} err="failed to get container status \"b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33\": rpc error: code = NotFound desc = could not find container \"b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33\": container with ID starting with b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33 not found: ID does not exist"
May 20 14:25:17.978698 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.978650    2550 scope.go:117] "RemoveContainer" containerID="d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b"
May 20 14:25:17.978925 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:25:17.978911    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b\": container with ID starting with d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b not found: ID does not exist" containerID="d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b"
May 20 14:25:17.978964 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.978929    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b"} err="failed to get container status \"d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b\": rpc error: code = NotFound desc = could not find container \"d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b\": container with ID starting with d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b not found: ID does not exist"
May 20 14:25:17.978964 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.978948    2550 scope.go:117] "RemoveContainer" containerID="d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586"
May 20 14:25:17.979182 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:25:17.979161    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586\": container with ID starting with d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586 not found: ID does not exist" containerID="d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586"
May 20 14:25:17.979229 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.979188    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586"} err="failed to get container status \"d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586\": rpc error: code = NotFound desc = could not find container \"d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586\": container with ID starting with d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586 not found: ID does not exist"
May 20 14:25:17.979229 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.979205    2550 scope.go:117] "RemoveContainer" containerID="b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33"
May 20 14:25:17.979442 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.979418    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33"} err="failed to get container status \"b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33\": rpc error: code = NotFound desc = could not find container \"b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33\": container with ID starting with b76631e26e50bd9cbf6f7af999a3f3ed7d90a7c51570c4ede87927081145ca33 not found: ID does not exist"
May 20 14:25:17.979527 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.979443    2550 scope.go:117] "RemoveContainer" containerID="d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b"
May 20 14:25:17.979678 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.979660    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b"} err="failed to get container status \"d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b\": rpc error: code = NotFound desc = could not find container \"d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b\": container with ID starting with d46407d267af4dd7007a7a153a02b24604f2040c2f0eab5f5a0ce98a1933662b not found: ID does not exist"
May 20 14:25:17.979734 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.979679    2550 scope.go:117] "RemoveContainer" containerID="d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586"
May 20 14:25:17.979888 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:17.979861    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586"} err="failed to get container status \"d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586\": rpc error: code = NotFound desc = could not find container \"d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586\": container with ID starting with d716f70ede2b5e4b3de46974e1f92c655e695b22345b54bd61ba9ceb7580a586 not found: ID does not exist"
May 20 14:25:18.467349 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:18.467250    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" path="/var/lib/kubelet/pods/58817abd-ef34-43f6-84dc-f2a36b788fc0/volumes"
May 20 14:25:18.467794 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:18.467778    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" path="/var/lib/kubelet/pods/8982c4e0-0df7-44cb-9787-a7f65f289e46/volumes"
May 20 14:25:21.171355 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:21.171307    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
May 20 14:25:31.171138 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:31.171088    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
May 20 14:25:41.171700 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:41.171654    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
May 20 14:25:51.171629 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:25:51.171580    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
May 20 14:26:01.171376 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:01.171321    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
May 20 14:26:11.170960 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:11.170915    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
May 20 14:26:21.171444 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:21.171397    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
May 20 14:26:31.171209 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:31.171162    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
May 20 14:26:41.181521 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:41.181485    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:26:41.189659 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:41.189625    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:26:46.155738 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:46.155704    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"]
May 20 14:26:46.156142 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:46.155963    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main" containerID="cri-o://828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc" gracePeriod=30
May 20 14:26:59.243010 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.242973    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243309    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="llm-d-routing-sidecar"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243320    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="llm-d-routing-sidecar"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243348    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243354    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243373    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="storage-initializer"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243379    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="storage-initializer"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243385    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="storage-initializer"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243390    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="storage-initializer"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243399    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243410    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243461    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="llm-d-routing-sidecar"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243472    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="8982c4e0-0df7-44cb-9787-a7f65f289e46" containerName="main"
May 20 14:26:59.243505 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.243480    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="58817abd-ef34-43f6-84dc-f2a36b788fc0" containerName="main"
May 20 14:26:59.247114 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.247092    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.250040 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.250017    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 20 14:26:59.250167 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.250129    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-96lmg\""
May 20 14:26:59.255957 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.255932    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 14:26:59.378739 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.378696    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25tq4\" (UniqueName: \"kubernetes.io/projected/230fb49d-1996-464b-99f5-ee32742e188c-kube-api-access-25tq4\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.378739 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.378743    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.378973 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.378772    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.378973 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.378817    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230fb49d-1996-464b-99f5-ee32742e188c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.378973 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.378855    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.378973 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.378894    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.378973 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.378936    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.479810 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.479770    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-25tq4\" (UniqueName: \"kubernetes.io/projected/230fb49d-1996-464b-99f5-ee32742e188c-kube-api-access-25tq4\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.479810 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.479816    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.480029 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.479835    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.480029 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.479955    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230fb49d-1996-464b-99f5-ee32742e188c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.480029 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.480005    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.480133 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.480069    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.480133 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.480108    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.480211 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.480156    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.480211 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.480197    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.480373 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.480323    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.480421 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.480363    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.482293 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.482265    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.482453 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.482436    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230fb49d-1996-464b-99f5-ee32742e188c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.487268 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.487245    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25tq4\" (UniqueName: \"kubernetes.io/projected/230fb49d-1996-464b-99f5-ee32742e188c-kube-api-access-25tq4\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.560065 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.560028    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:26:59.694413 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:26:59.694385    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 14:26:59.697158 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:26:59.697131    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230fb49d_1996_464b_99f5_ee32742e188c.slice/crio-38726e56a61607347c4d98c2474ed630c944b07ccbd67c1511bc5963d4ea6e25 WatchSource:0}: Error finding container 38726e56a61607347c4d98c2474ed630c944b07ccbd67c1511bc5963d4ea6e25: Status 404 returned error can't find the container with id 38726e56a61607347c4d98c2474ed630c944b07ccbd67c1511bc5963d4ea6e25
May 20 14:27:00.224173 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:00.224136    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"230fb49d-1996-464b-99f5-ee32742e188c","Type":"ContainerStarted","Data":"e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380"}
May 20 14:27:00.224412 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:00.224181    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"230fb49d-1996-464b-99f5-ee32742e188c","Type":"ContainerStarted","Data":"38726e56a61607347c4d98c2474ed630c944b07ccbd67c1511bc5963d4ea6e25"}
May 20 14:27:04.238887 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:04.238851    2550 generic.go:358] "Generic (PLEG): container finished" podID="230fb49d-1996-464b-99f5-ee32742e188c" containerID="e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380" exitCode=0
May 20 14:27:04.239389 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:04.238927    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"230fb49d-1996-464b-99f5-ee32742e188c","Type":"ContainerDied","Data":"e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380"}
May 20 14:27:05.244096 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:05.244058    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"230fb49d-1996-464b-99f5-ee32742e188c","Type":"ContainerStarted","Data":"df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14"}
May 20 14:27:05.264740 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:05.264677    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=6.264655475 podStartE2EDuration="6.264655475s" podCreationTimestamp="2026-05-20 14:26:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:27:05.262611479 +0000 UTC m=+2203.388994633" watchObservedRunningTime="2026-05-20 14:27:05.264655475 +0000 UTC m=+2203.391038629"
May 20 14:27:09.560520 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:09.560473    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:27:09.562075 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:09.562045    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
May 20 14:27:16.455931 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.455901    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc_7e8bd313-d746-4635-86c1-1c31adcf806d/main/0.log"
May 20 14:27:16.456320 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.456307    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:27:16.542708 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.542674    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e8bd313-d746-4635-86c1-1c31adcf806d-tls-certs\") pod \"7e8bd313-d746-4635-86c1-1c31adcf806d\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") "
May 20 14:27:16.542708 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.542709    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-home\") pod \"7e8bd313-d746-4635-86c1-1c31adcf806d\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") "
May 20 14:27:16.542969 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.542738    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-model-cache\") pod \"7e8bd313-d746-4635-86c1-1c31adcf806d\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") "
May 20 14:27:16.542969 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.542884    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-dshm\") pod \"7e8bd313-d746-4635-86c1-1c31adcf806d\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") "
May 20 14:27:16.542969 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.542939    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-tmp-dir\") pod \"7e8bd313-d746-4635-86c1-1c31adcf806d\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") "
May 20 14:27:16.543122 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.542987    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-kserve-provision-location\") pod \"7e8bd313-d746-4635-86c1-1c31adcf806d\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") "
May 20 14:27:16.543122 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.543013    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e8bd313-d746-4635-86c1-1c31adcf806d" (UID: "7e8bd313-d746-4635-86c1-1c31adcf806d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:27:16.543122 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.543040    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68pct\" (UniqueName: \"kubernetes.io/projected/7e8bd313-d746-4635-86c1-1c31adcf806d-kube-api-access-68pct\") pod \"7e8bd313-d746-4635-86c1-1c31adcf806d\" (UID: \"7e8bd313-d746-4635-86c1-1c31adcf806d\") "
May 20 14:27:16.543394 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.543361    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:27:16.543580 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.543552    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-home" (OuterVolumeSpecName: "home") pod "7e8bd313-d746-4635-86c1-1c31adcf806d" (UID: "7e8bd313-d746-4635-86c1-1c31adcf806d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:27:16.545557 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.545518    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-dshm" (OuterVolumeSpecName: "dshm") pod "7e8bd313-d746-4635-86c1-1c31adcf806d" (UID: "7e8bd313-d746-4635-86c1-1c31adcf806d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:27:16.545678 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.545638    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8bd313-d746-4635-86c1-1c31adcf806d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e8bd313-d746-4635-86c1-1c31adcf806d" (UID: "7e8bd313-d746-4635-86c1-1c31adcf806d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:27:16.545925 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.545894    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8bd313-d746-4635-86c1-1c31adcf806d-kube-api-access-68pct" (OuterVolumeSpecName: "kube-api-access-68pct") pod "7e8bd313-d746-4635-86c1-1c31adcf806d" (UID: "7e8bd313-d746-4635-86c1-1c31adcf806d"). InnerVolumeSpecName "kube-api-access-68pct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:27:16.558488 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.558450    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e8bd313-d746-4635-86c1-1c31adcf806d" (UID: "7e8bd313-d746-4635-86c1-1c31adcf806d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:27:16.604229 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.604170    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e8bd313-d746-4635-86c1-1c31adcf806d" (UID: "7e8bd313-d746-4635-86c1-1c31adcf806d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:27:16.644125 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.644086    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68pct\" (UniqueName: \"kubernetes.io/projected/7e8bd313-d746-4635-86c1-1c31adcf806d-kube-api-access-68pct\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:27:16.644125 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.644120    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e8bd313-d746-4635-86c1-1c31adcf806d-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:27:16.644125 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.644130    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:27:16.644386 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.644139    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:27:16.644386 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.644147    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:27:16.644386 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:16.644155    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e8bd313-d746-4635-86c1-1c31adcf806d-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:27:17.285697 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.285669    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc_7e8bd313-d746-4635-86c1-1c31adcf806d/main/0.log"
May 20 14:27:17.286032 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.286000    2550 generic.go:358] "Generic (PLEG): container finished" podID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerID="828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc" exitCode=137
May 20 14:27:17.286145 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.286070    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" event={"ID":"7e8bd313-d746-4635-86c1-1c31adcf806d","Type":"ContainerDied","Data":"828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc"}
May 20 14:27:17.286145 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.286078    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"
May 20 14:27:17.286145 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.286102    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc" event={"ID":"7e8bd313-d746-4635-86c1-1c31adcf806d","Type":"ContainerDied","Data":"941d49a614f11713420b591498f01b03a13c01fcf1cb31fbf0766ee969e824b6"}
May 20 14:27:17.286145 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.286117    2550 scope.go:117] "RemoveContainer" containerID="828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc"
May 20 14:27:17.297915 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.297587    2550 scope.go:117] "RemoveContainer" containerID="975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d"
May 20 14:27:17.309641 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.309605    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"]
May 20 14:27:17.311019 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.310853    2550 scope.go:117] "RemoveContainer" containerID="828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc"
May 20 14:27:17.312462 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:27:17.311591    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc\": container with ID starting with 828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc not found: ID does not exist" containerID="828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc"
May 20 14:27:17.312462 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.311635    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc"} err="failed to get container status \"828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc\": rpc error: code = NotFound desc = could not find container \"828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc\": container with ID starting with 828d7d7bf4453fdc66e273a3915958b373c2f519f44ad6237dbfba447616f3bc not found: ID does not exist"
May 20 14:27:17.312462 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.311663    2550 scope.go:117] "RemoveContainer" containerID="975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d"
May 20 14:27:17.312462 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:27:17.312387    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d\": container with ID starting with 975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d not found: ID does not exist" containerID="975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d"
May 20 14:27:17.312462 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.312420    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d"} err="failed to get container status \"975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d\": rpc error: code = NotFound desc = could not find container \"975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d\": container with ID starting with 975b592abc276f8de37442085796f448074ba64db26c9cbd1f55bf72513f586d not found: ID does not exist"
May 20 14:27:17.313379 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:17.313356    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-6fdc6f5f95t7rxc"]
May 20 14:27:18.467571 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:18.467537    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" path="/var/lib/kubelet/pods/7e8bd313-d746-4635-86c1-1c31adcf806d/volumes"
May 20 14:27:19.561298 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:19.561240    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
May 20 14:27:29.560566 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:29.560525    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:27:29.561038 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:29.560867    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
May 20 14:27:39.561555 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:39.561495    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
May 20 14:27:49.561031 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:49.560985    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
May 20 14:27:59.561174 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:27:59.561111    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
May 20 14:28:09.560811 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:09.560769    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
May 20 14:28:19.560933 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:19.560880    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
May 20 14:28:29.561379 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:29.561308    2550 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
May 20 14:28:39.575989 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:39.575955    2550 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:28:39.583915 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:39.583891    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:28:47.670723 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:47.670686    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 14:28:47.671258 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:47.670979    2550 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main" containerID="cri-o://df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14" gracePeriod=30
May 20 14:28:48.431699 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.431673    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:28:48.481749 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.481666    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-kserve-provision-location\") pod \"230fb49d-1996-464b-99f5-ee32742e188c\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") "
May 20 14:28:48.481749 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.481699    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-model-cache\") pod \"230fb49d-1996-464b-99f5-ee32742e188c\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") "
May 20 14:28:48.481749 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.481746    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-tmp-dir\") pod \"230fb49d-1996-464b-99f5-ee32742e188c\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") "
May 20 14:28:48.482026 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.481795    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25tq4\" (UniqueName: \"kubernetes.io/projected/230fb49d-1996-464b-99f5-ee32742e188c-kube-api-access-25tq4\") pod \"230fb49d-1996-464b-99f5-ee32742e188c\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") "
May 20 14:28:48.482026 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.481822    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-dshm\") pod \"230fb49d-1996-464b-99f5-ee32742e188c\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") "
May 20 14:28:48.482026 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.481850    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230fb49d-1996-464b-99f5-ee32742e188c-tls-certs\") pod \"230fb49d-1996-464b-99f5-ee32742e188c\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") "
May 20 14:28:48.482026 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.481898    2550 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-home\") pod \"230fb49d-1996-464b-99f5-ee32742e188c\" (UID: \"230fb49d-1996-464b-99f5-ee32742e188c\") "
May 20 14:28:48.482240 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.482031    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-model-cache" (OuterVolumeSpecName: "model-cache") pod "230fb49d-1996-464b-99f5-ee32742e188c" (UID: "230fb49d-1996-464b-99f5-ee32742e188c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:48.482240 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.482233    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-model-cache\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:28:48.482722 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.482676    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-home" (OuterVolumeSpecName: "home") pod "230fb49d-1996-464b-99f5-ee32742e188c" (UID: "230fb49d-1996-464b-99f5-ee32742e188c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:48.484571 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.484535    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-dshm" (OuterVolumeSpecName: "dshm") pod "230fb49d-1996-464b-99f5-ee32742e188c" (UID: "230fb49d-1996-464b-99f5-ee32742e188c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:48.484687 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.484612    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230fb49d-1996-464b-99f5-ee32742e188c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "230fb49d-1996-464b-99f5-ee32742e188c" (UID: "230fb49d-1996-464b-99f5-ee32742e188c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 14:28:48.484749 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.484702    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230fb49d-1996-464b-99f5-ee32742e188c-kube-api-access-25tq4" (OuterVolumeSpecName: "kube-api-access-25tq4") pod "230fb49d-1996-464b-99f5-ee32742e188c" (UID: "230fb49d-1996-464b-99f5-ee32742e188c"). InnerVolumeSpecName "kube-api-access-25tq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 14:28:48.495912 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.495851    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "230fb49d-1996-464b-99f5-ee32742e188c" (UID: "230fb49d-1996-464b-99f5-ee32742e188c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:48.542551 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.542508    2550 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "230fb49d-1996-464b-99f5-ee32742e188c" (UID: "230fb49d-1996-464b-99f5-ee32742e188c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 14:28:48.583383 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.583315    2550 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-kserve-provision-location\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:28:48.583383 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.583378    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-tmp-dir\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:28:48.583383 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.583393    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25tq4\" (UniqueName: \"kubernetes.io/projected/230fb49d-1996-464b-99f5-ee32742e188c-kube-api-access-25tq4\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:28:48.583643 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.583409    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-dshm\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:28:48.583643 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.583419    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230fb49d-1996-464b-99f5-ee32742e188c-tls-certs\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:28:48.583643 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.583427    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/230fb49d-1996-464b-99f5-ee32742e188c-home\") on node \"ip-10-0-129-117.ec2.internal\" DevicePath \"\""
May 20 14:28:48.610621 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.610576    2550 generic.go:358] "Generic (PLEG): container finished" podID="230fb49d-1996-464b-99f5-ee32742e188c" containerID="df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14" exitCode=0
May 20 14:28:48.610822 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.610623    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"230fb49d-1996-464b-99f5-ee32742e188c","Type":"ContainerDied","Data":"df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14"}
May 20 14:28:48.610822 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.610659    2550 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 14:28:48.610822 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.610667    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"230fb49d-1996-464b-99f5-ee32742e188c","Type":"ContainerDied","Data":"38726e56a61607347c4d98c2474ed630c944b07ccbd67c1511bc5963d4ea6e25"}
May 20 14:28:48.610822 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.610689    2550 scope.go:117] "RemoveContainer" containerID="df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14"
May 20 14:28:48.619440 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.619413    2550 scope.go:117] "RemoveContainer" containerID="e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380"
May 20 14:28:48.629366 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.629345    2550 scope.go:117] "RemoveContainer" containerID="df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14"
May 20 14:28:48.629648 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:28:48.629628    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14\": container with ID starting with df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14 not found: ID does not exist" containerID="df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14"
May 20 14:28:48.629725 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.629656    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14"} err="failed to get container status \"df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14\": rpc error: code = NotFound desc = could not find container \"df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14\": container with ID starting with df2afac31671d4c1165ef5be2124909b590b9bbcce6c43b96496beebaf103c14 not found: ID does not exist"
May 20 14:28:48.629725 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.629676    2550 scope.go:117] "RemoveContainer" containerID="e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380"
May 20 14:28:48.629931 ip-10-0-129-117 kubenswrapper[2550]: E0520 14:28:48.629913    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380\": container with ID starting with e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380 not found: ID does not exist" containerID="e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380"
May 20 14:28:48.629969 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.629936    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380"} err="failed to get container status \"e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380\": rpc error: code = NotFound desc = could not find container \"e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380\": container with ID starting with e3474d19a1fc0ab65fe737046628d1f54bf55637d8459c6806c8f1533b54a380 not found: ID does not exist"
May 20 14:28:48.634239 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.634209    2550 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 14:28:48.639953 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:48.639927    2550 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 14:28:50.467723 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:28:50.467689    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230fb49d-1996-464b-99f5-ee32742e188c" path="/var/lib/kubelet/pods/230fb49d-1996-464b-99f5-ee32742e188c/volumes"
May 20 14:29:04.027989 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:04.027955    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:05.046728 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:05.046694    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:06.054467 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:06.054437    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:07.026918 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:07.026875    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:08.016594 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:08.016564    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:09.055279 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:09.055244    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:10.026599 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:10.026554    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:10.995559 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:10.995526    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:11.964841 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:11.964763    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:12.937407 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:12.937375    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:13.909457 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:13.909425    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:14.883573 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:14.883540    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:15.877201 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:15.877172    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:16.886730 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:16.886699    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-96dfd8dc5-2rrzb_4dfc6c23-ee5b-4734-9325-bac9b9815fe2/storage-initializer/0.log"
May 20 14:29:17.984493 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:17.984440    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-mncdp_f825add0-7ef6-4b98-9794-ee03badebc8f/discovery/0.log"
May 20 14:29:18.018513 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:18.018475    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656dc888b4-j7s2n_daad92a8-291c-4a57-9bb3-40d58be05740/router/0.log"
May 20 14:29:18.842139 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:18.842101    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-mncdp_f825add0-7ef6-4b98-9794-ee03badebc8f/discovery/0.log"
May 20 14:29:18.902567 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:18.902537    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656dc888b4-j7s2n_daad92a8-291c-4a57-9bb3-40d58be05740/router/0.log"
May 20 14:29:19.690496 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:19.690461    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-mncdp_f825add0-7ef6-4b98-9794-ee03badebc8f/discovery/0.log"
May 20 14:29:19.717729 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:19.717703    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656dc888b4-j7s2n_daad92a8-291c-4a57-9bb3-40d58be05740/router/0.log"
May 20 14:29:20.455148 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:20.455122    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-r4lmp_d838655f-8373-4ecf-9eb0-8d3a5572835b/kuadrant-console-plugin/0.log"
May 20 14:29:21.344473 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:21.344442    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-r4lmp_d838655f-8373-4ecf-9eb0-8d3a5572835b/kuadrant-console-plugin/0.log"
May 20 14:29:22.276424 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:22.276394    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-r4lmp_d838655f-8373-4ecf-9eb0-8d3a5572835b/kuadrant-console-plugin/0.log"
May 20 14:29:23.197507 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:23.197481    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-r4lmp_d838655f-8373-4ecf-9eb0-8d3a5572835b/kuadrant-console-plugin/0.log"
May 20 14:29:24.085430 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:24.085399    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-r4lmp_d838655f-8373-4ecf-9eb0-8d3a5572835b/kuadrant-console-plugin/0.log"
May 20 14:29:29.253874 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:29.253842    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jnqxh_e7efcced-743d-485e-a499-25e831588966/global-pull-secret-syncer/0.log"
May 20 14:29:29.390603 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:29.390567    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t2fbl_5f08a998-c27c-422e-bfe3-697e2be370d2/konnectivity-agent/0.log"
May 20 14:29:29.413095 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:29.413065    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-117.ec2.internal_6f81a807c436a78d0ae12cc5e66b2e05/haproxy/0.log"
May 20 14:29:33.803263 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:33.803230    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-r4lmp_d838655f-8373-4ecf-9eb0-8d3a5572835b/kuadrant-console-plugin/0.log"
May 20 14:29:34.965294 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:34.965258    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_788c6eee-255d-48dd-85aa-53a51ed02d73/alertmanager/0.log"
May 20 14:29:34.986288 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:34.986253    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_788c6eee-255d-48dd-85aa-53a51ed02d73/config-reloader/0.log"
May 20 14:29:35.009032 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.008997    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_788c6eee-255d-48dd-85aa-53a51ed02d73/kube-rbac-proxy-web/0.log"
May 20 14:29:35.030330 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.030300    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_788c6eee-255d-48dd-85aa-53a51ed02d73/kube-rbac-proxy/0.log"
May 20 14:29:35.051515 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.051493    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_788c6eee-255d-48dd-85aa-53a51ed02d73/kube-rbac-proxy-metric/0.log"
May 20 14:29:35.071483 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.071454    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_788c6eee-255d-48dd-85aa-53a51ed02d73/prom-label-proxy/0.log"
May 20 14:29:35.092999 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.092972    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_788c6eee-255d-48dd-85aa-53a51ed02d73/init-config-reloader/0.log"
May 20 14:29:35.137860 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.137824    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-2xrcl_74f1e9a4-b0ef-48e5-8107-1b9a218414e1/cluster-monitoring-operator/0.log"
May 20 14:29:35.163173 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.163134    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-dtrm7_05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9/kube-state-metrics/0.log"
May 20 14:29:35.179007 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.178979    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-dtrm7_05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9/kube-rbac-proxy-main/0.log"
May 20 14:29:35.198147 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.198125    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-dtrm7_05c9bb3e-e318-4dcb-bcc3-832ff3f6d4a9/kube-rbac-proxy-self/0.log"
May 20 14:29:35.222544 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.222438    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7478978558-hqpxf_fa74be60-54d6-45f8-b61a-2109c576facb/metrics-server/0.log"
May 20 14:29:35.356776 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.356746    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dmbqj_7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79/node-exporter/0.log"
May 20 14:29:35.374923 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.374891    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dmbqj_7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79/kube-rbac-proxy/0.log"
May 20 14:29:35.397239 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.397215    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dmbqj_7d3ff7ba-5b0b-4ad3-ad8c-e6c0da52ab79/init-textfile/0.log"
May 20 14:29:35.503537 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.503506    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-pmb5p_6a17c27f-6c21-4883-92a4-00888af609ad/kube-rbac-proxy-main/0.log"
May 20 14:29:35.526739 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.526705    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-pmb5p_6a17c27f-6c21-4883-92a4-00888af609ad/kube-rbac-proxy-self/0.log"
May 20 14:29:35.547255 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.547223    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-pmb5p_6a17c27f-6c21-4883-92a4-00888af609ad/openshift-state-metrics/0.log"
May 20 14:29:35.758576 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.758502    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-tsjd8_b9411521-d5d0-45c4-97e0-1c6cc722e20d/prometheus-operator/0.log"
May 20 14:29:35.779678 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.779656    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-tsjd8_b9411521-d5d0-45c4-97e0-1c6cc722e20d/kube-rbac-proxy/0.log"
May 20 14:29:35.853716 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.853684    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f9869f974-6pb2h_13f70248-994e-4682-af6b-8e6cc7c863c0/telemeter-client/0.log"
May 20 14:29:35.877480 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.877453    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f9869f974-6pb2h_13f70248-994e-4682-af6b-8e6cc7c863c0/reload/0.log"
May 20 14:29:35.895127 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:35.895100    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f9869f974-6pb2h_13f70248-994e-4682-af6b-8e6cc7c863c0/kube-rbac-proxy/0.log"
May 20 14:29:38.194397 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.194360    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-gg6jn_74dd87a5-973e-4409-a688-1f1e5c0f840f/download-server/0.log"
May 20 14:29:38.563627 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.563593    2550 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"]
May 20 14:29:38.564386 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.564093    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main"
May 20 14:29:38.564386 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.564117    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main"
May 20 14:29:38.564386 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.564144    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main"
May 20 14:29:38.564386 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.564152    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main"
May 20 14:29:38.564386 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.564173    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="storage-initializer"
May 20 14:29:38.564386 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.564182    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="storage-initializer"
May 20 14:29:38.564386 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.564193    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="storage-initializer"
May 20 14:29:38.564386 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.564202    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="storage-initializer"
May 20 14:29:38.564760 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.564462    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="7e8bd313-d746-4635-86c1-1c31adcf806d" containerName="main"
May 20 14:29:38.564760 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.564478    2550 memory_manager.go:356] "RemoveStaleState removing state" podUID="230fb49d-1996-464b-99f5-ee32742e188c" containerName="main"
May 20 14:29:38.568742 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.568724    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.571813 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.571760    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cpmvk\"/\"kube-root-ca.crt\""
May 20 14:29:38.571813 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.571782    2550 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cpmvk\"/\"openshift-service-ca.crt\""
May 20 14:29:38.572734 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.572718    2550 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cpmvk\"/\"default-dockercfg-rh55f\""
May 20 14:29:38.576638 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.576614    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"]
May 20 14:29:38.623142 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.623092    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-proc\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.623329 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.623182    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-lib-modules\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.623329 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.623208    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl5xj\" (UniqueName: \"kubernetes.io/projected/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-kube-api-access-dl5xj\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.623329 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.623234    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-podres\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.623329 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.623253    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-sys\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.645822 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.645795    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-nddf5_4baead3c-57b6-4b3d-966b-daf23038fa89/volume-data-source-validator/0.log"
May 20 14:29:38.724020 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.723978    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-lib-modules\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.724204 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.724027    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dl5xj\" (UniqueName: \"kubernetes.io/projected/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-kube-api-access-dl5xj\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.724204 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.724055    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-podres\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.724204 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.724072    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-sys\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.724204 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.724121    2550 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-proc\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.724204 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.724184    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-lib-modules\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.724446 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.724184    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-podres\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.724446 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.724224    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-proc\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.724446 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.724224    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-sys\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.732638 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.732609    2550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl5xj\" (UniqueName: \"kubernetes.io/projected/bf04993d-b0b9-4fda-b42f-b4fee1b1c62d-kube-api-access-dl5xj\") pod \"perf-node-gather-daemonset-84vgj\" (UID: \"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d\") " pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:38.880493 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:38.880400    2550 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:39.012434 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:39.012408    2550 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"]
May 20 14:29:39.014647 ip-10-0-129-117 kubenswrapper[2550]: W0520 14:29:39.014616    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbf04993d_b0b9_4fda_b42f_b4fee1b1c62d.slice/crio-8676f270d39e353c5af4bff2ae278b09a28ef8d249c5519b1902a6229f7bf77b WatchSource:0}: Error finding container 8676f270d39e353c5af4bff2ae278b09a28ef8d249c5519b1902a6229f7bf77b: Status 404 returned error can't find the container with id 8676f270d39e353c5af4bff2ae278b09a28ef8d249c5519b1902a6229f7bf77b
May 20 14:29:39.391014 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:39.390931    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lxrtg_e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6/dns/0.log"
May 20 14:29:39.409894 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:39.409862    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lxrtg_e459bac0-e5e7-41b0-b9c4-6d6e49d6cfe6/kube-rbac-proxy/0.log"
May 20 14:29:39.555238 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:39.555208    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rcsnb_576d8a0a-be91-4189-b61a-3cec8a8834f2/dns-node-resolver/0.log"
May 20 14:29:39.788809 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:39.788777    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj" event={"ID":"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d","Type":"ContainerStarted","Data":"4134f99fbc85521d36119469ff049f23d4a4f690265511307013228ff1e1b906"}
May 20 14:29:39.788809 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:39.788811    2550 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj" event={"ID":"bf04993d-b0b9-4fda-b42f-b4fee1b1c62d","Type":"ContainerStarted","Data":"8676f270d39e353c5af4bff2ae278b09a28ef8d249c5519b1902a6229f7bf77b"}
May 20 14:29:39.789072 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:39.788846    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:39.805528 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:39.805478    2550 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj" podStartSLOduration=1.80545903 podStartE2EDuration="1.80545903s" podCreationTimestamp="2026-05-20 14:29:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 14:29:39.804330309 +0000 UTC m=+2357.930713475" watchObservedRunningTime="2026-05-20 14:29:39.80545903 +0000 UTC m=+2357.931842184"
May 20 14:29:40.024457 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:40.024424    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2gc7l_27731076-9ec4-4347-8b2f-0baf5b1e8622/node-ca/0.log"
May 20 14:29:40.896507 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:40.896468    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-mncdp_f825add0-7ef6-4b98-9794-ee03badebc8f/discovery/0.log"
May 20 14:29:40.944784 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:40.944752    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656dc888b4-j7s2n_daad92a8-291c-4a57-9bb3-40d58be05740/router/0.log"
May 20 14:29:41.373479 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:41.373451    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2vfk7_374503b2-0a1d-41f6-9fc7-b81f43151c0e/serve-healthcheck-canary/0.log"
May 20 14:29:41.965270 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:41.965240    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkftb_923ac251-4722-4870-be54-9843ac346d9f/kube-rbac-proxy/0.log"
May 20 14:29:41.983566 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:41.983539    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkftb_923ac251-4722-4870-be54-9843ac346d9f/exporter/0.log"
May 20 14:29:42.002138 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:42.002110    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkftb_923ac251-4722-4870-be54-9843ac346d9f/extractor/0.log"
May 20 14:29:44.520045 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:44.520012    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-79d5949c76-nccdr_a5439963-59db-4d32-a3d9-aa94fa511462/manager/0.log"
May 20 14:29:44.542402 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:44.542359    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-bfc7f696d-r8xpj_b90bc282-64a2-4236-be4a-885652b98896/openshift-lws-operator/0.log"
May 20 14:29:45.802079 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:45.802053    2550 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cpmvk/perf-node-gather-daemonset-84vgj"
May 20 14:29:51.444408 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:51.444365    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-49mt5_05155843-11d8-40f1-ad79-012c0cd08e83/kube-multus/0.log"
May 20 14:29:51.761245 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:51.761219    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qx4t2_6502c66b-dfda-470b-98f8-6ec092bcfec2/kube-multus-additional-cni-plugins/0.log"
May 20 14:29:51.779553 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:51.779520    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qx4t2_6502c66b-dfda-470b-98f8-6ec092bcfec2/egress-router-binary-copy/0.log"
May 20 14:29:51.798281 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:51.798255    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qx4t2_6502c66b-dfda-470b-98f8-6ec092bcfec2/cni-plugins/0.log"
May 20 14:29:51.817096 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:51.817066    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qx4t2_6502c66b-dfda-470b-98f8-6ec092bcfec2/bond-cni-plugin/0.log"
May 20 14:29:51.836369 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:51.836323    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qx4t2_6502c66b-dfda-470b-98f8-6ec092bcfec2/routeoverride-cni/0.log"
May 20 14:29:51.857030 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:51.857005    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qx4t2_6502c66b-dfda-470b-98f8-6ec092bcfec2/whereabouts-cni-bincopy/0.log"
May 20 14:29:51.874756 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:51.874729    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qx4t2_6502c66b-dfda-470b-98f8-6ec092bcfec2/whereabouts-cni/0.log"
May 20 14:29:52.013359 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:52.013258    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rvjzj_a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be/network-metrics-daemon/0.log"
May 20 14:29:52.031965 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:52.031944    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rvjzj_a281ce9a-c6ff-4bbb-a2bc-ae0ed33679be/kube-rbac-proxy/0.log"
May 20 14:29:52.872865 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:52.872832    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbcdp_b7b7bb27-c274-4fa1-a5fb-97fe855429c3/ovn-controller/0.log"
May 20 14:29:52.907674 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:52.907642    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbcdp_b7b7bb27-c274-4fa1-a5fb-97fe855429c3/ovn-acl-logging/0.log"
May 20 14:29:52.932552 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:52.932525    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbcdp_b7b7bb27-c274-4fa1-a5fb-97fe855429c3/kube-rbac-proxy-node/0.log"
May 20 14:29:52.951030 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:52.951003    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbcdp_b7b7bb27-c274-4fa1-a5fb-97fe855429c3/kube-rbac-proxy-ovn-metrics/0.log"
May 20 14:29:52.966967 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:52.966944    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbcdp_b7b7bb27-c274-4fa1-a5fb-97fe855429c3/northd/0.log"
May 20 14:29:52.984347 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:52.984315    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbcdp_b7b7bb27-c274-4fa1-a5fb-97fe855429c3/nbdb/0.log"
May 20 14:29:53.002726 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:53.002703    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbcdp_b7b7bb27-c274-4fa1-a5fb-97fe855429c3/sbdb/0.log"
May 20 14:29:53.178563 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:53.178482    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jbcdp_b7b7bb27-c274-4fa1-a5fb-97fe855429c3/ovnkube-controller/0.log"
May 20 14:29:54.830265 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:54.830238    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-7hds7_8d2b9089-b3ec-4941-91cc-7c3cada6db01/network-check-target-container/0.log"
May 20 14:29:55.843198 ip-10-0-129-117 kubenswrapper[2550]: I0520 14:29:55.843172    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-th9rk_148ba522-00f2-49f0-b1b2-0d3c8860b2eb/iptables-alerter/0.log"