May 20 15:29:15.685338 ip-10-0-130-227 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 20 15:29:15.685512 ip-10-0-130-227 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 20 15:29:15.685614 ip-10-0-130-227 systemd[1]: kubelet.service: Failed with result 'resources'.
May 20 15:29:15.686313 ip-10-0-130-227 systemd[1]: Failed to start Kubernetes Kubelet.
May 20 15:29:26.906975 ip-10-0-130-227 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 20 15:29:26.906990 ip-10-0-130-227 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bf70e3f44b9448ccb8bec5c4f8803ae7 --
May 20 15:31:50.147917 ip-10-0-130-227 systemd[1]: Starting Kubernetes Kubelet...
May 20 15:31:50.788622 ip-10-0-130-227 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:50.788622 ip-10-0-130-227 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 20 15:31:50.788622 ip-10-0-130-227 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:50.788622 ip-10-0-130-227 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 20 15:31:50.788622 ip-10-0-130-227 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 20 15:31:50.789878 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.789782    2567 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 20 15:31:50.799238 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799215    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:50.799238 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799234    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:50.799238 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799237    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:50.799238 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799241    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799244    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799248    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799250    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799253    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799256    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799258    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799261    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799263    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799266    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799268    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799271    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799273    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799277    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799286    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799289    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799291    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799295    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799299    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799302    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:50.799366 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799305    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799307    2567 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799310    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799312    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799315    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799317    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799320    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799323    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799325    2567 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799328    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799331    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799333    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799335    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799338    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799342    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799344    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799347    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799350    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799353    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799356    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:50.799855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799358    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799361    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799364    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799367    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799369    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799372    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799374    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799377    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799380    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799383    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799385    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799388    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799391    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799394    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799397    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799399    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799402    2567 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799404    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799406    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799409    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:50.800354 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799411    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799415    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799419    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799429    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799431    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799436    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799439    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799442    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799444    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799447    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799450    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799452    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799455    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799457    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799460    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799463    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799465    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799469    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799471    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:50.800870 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799474    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799476    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799479    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799481    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799923    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799930    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799933    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799936    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799939    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799942    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799945    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799948    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799951    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799954    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799957    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799959    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799963    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799966    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799970    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:50.801321 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799973    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799975    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799978    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799981    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799983    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799986    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799988    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799991    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799994    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799996    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.799999    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800001    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800003    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800006    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800008    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800011    2567 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800014    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800017    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800019    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800022    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:50.801785 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800024    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800028    2567 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800030    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800032    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800035    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800037    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800040    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800042    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800044    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800047    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800050    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800053    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800056    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800058    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800061    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800064    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800066    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800069    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800071    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800074    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:50.802266 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800076    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800079    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800081    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800084    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800086    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800089    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800091    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800094    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800096    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800099    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800101    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800103    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800106    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800108    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800112    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800116    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800119    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800121    2567 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800124    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800126    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:50.802781 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800129    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800131    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800134    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800141    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800144    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800146    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800149    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800151    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800153    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800156    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.800158    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801337    2567 flags.go:64] FLAG: --address="0.0.0.0"
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801347    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801353    2567 flags.go:64] FLAG: --anonymous-auth="true"
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801358    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801363    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801366    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801371    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801375    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801379    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801383    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 20 15:31:50.803261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801386    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801390    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801393    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801396    2567 flags.go:64] FLAG: --cgroup-root=""
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801399    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801402    2567 flags.go:64] FLAG: --client-ca-file=""
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801405    2567 flags.go:64] FLAG: --cloud-config=""
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801408    2567 flags.go:64] FLAG: --cloud-provider="external"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801411    2567 flags.go:64] FLAG: --cluster-dns="[]"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801416    2567 flags.go:64] FLAG: --cluster-domain=""
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801419    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801422    2567 flags.go:64] FLAG: --config-dir=""
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801424    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801428    2567 flags.go:64] FLAG: --container-log-max-files="5"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801433    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801436    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801440    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801443    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801446    2567 flags.go:64] FLAG: --contention-profiling="false"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801449    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801452    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801456    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801458    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801463    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801465    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 20 15:31:50.803775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801468    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801471    2567 flags.go:64] FLAG: --enable-load-reader="false"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801475    2567 flags.go:64] FLAG: --enable-server="true"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801478    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801482    2567 flags.go:64] FLAG: --event-burst="100"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801485    2567 flags.go:64] FLAG: --event-qps="50"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801488    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801491    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801494    2567 flags.go:64] FLAG: --eviction-hard=""
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801497    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801500    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801503    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801506    2567 flags.go:64] FLAG: --eviction-soft=""
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801509    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801512    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801515    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801518    2567 flags.go:64] FLAG: --experimental-mounter-path=""
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801521    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801528    2567 flags.go:64] FLAG: --fail-swap-on="true"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801531    2567 flags.go:64] FLAG: --feature-gates=""
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801535    2567 flags.go:64] FLAG: --file-check-frequency="20s"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801538    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801542    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801546    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801549    2567 flags.go:64] FLAG: --healthz-port="10248"
May 20 15:31:50.804411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801552    2567 flags.go:64] FLAG: --help="false"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801555    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-130-227.ec2.internal"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801558    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801561    2567 flags.go:64] FLAG: --http-check-frequency="20s"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801564    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801568    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801571    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801574    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801577    2567 flags.go:64] FLAG: --image-service-endpoint=""
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801594    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801597    2567 flags.go:64] FLAG: --kube-api-burst="100"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801600    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801603    2567 flags.go:64] FLAG: --kube-api-qps="50"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801606    2567 flags.go:64] FLAG: --kube-reserved=""
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801609    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801612    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801615    2567 flags.go:64] FLAG: --kubelet-cgroups=""
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801618    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801621    2567 flags.go:64] FLAG: --lock-file=""
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801624    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801627    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801630    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801635    2567 flags.go:64] FLAG: --log-json-split-stream="false"
May 20 15:31:50.805020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801638    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801641    2567 flags.go:64] FLAG: --log-text-split-stream="false"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801645    2567 flags.go:64] FLAG: --logging-format="text"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801648    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801652    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801654    2567 flags.go:64] FLAG: --manifest-url=""
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801657    2567 flags.go:64] FLAG: --manifest-url-header=""
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801662    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801665    2567 flags.go:64] FLAG: --max-open-files="1000000"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801669    2567 flags.go:64] FLAG: --max-pods="110"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801672    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801675    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801678    2567 flags.go:64] FLAG: --memory-manager-policy="None"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801681    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801685    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801688    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801691    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801698    2567 flags.go:64] FLAG: --node-status-max-images="50"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801702    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801705    2567 flags.go:64] FLAG: --oom-score-adj="-999"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801708    2567 flags.go:64] FLAG: --pod-cidr=""
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801711    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801717    2567 flags.go:64] FLAG: --pod-manifest-path=""
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801720    2567 flags.go:64] FLAG: --pod-max-pids="-1"
May 20 15:31:50.805657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801723    2567 flags.go:64] FLAG: --pods-per-core="0"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801726    2567 flags.go:64] FLAG: --port="10250"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801729    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801732    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d689fc635a0fae56"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801735    2567 flags.go:64] FLAG: --qos-reserved=""
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801737    2567 flags.go:64] FLAG: --read-only-port="10255"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801740    2567 flags.go:64] FLAG: --register-node="true"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801743    2567 flags.go:64] FLAG: --register-schedulable="true"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801746    2567 flags.go:64] FLAG: --register-with-taints=""
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801750    2567 flags.go:64] FLAG: --registry-burst="10"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801752    2567 flags.go:64] FLAG: --registry-qps="5"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801757    2567 flags.go:64] FLAG: --reserved-cpus=""
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801760    2567 flags.go:64] FLAG: --reserved-memory=""
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801764    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801767    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801771    2567 flags.go:64] FLAG: --rotate-certificates="false"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801773    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801777    2567 flags.go:64] FLAG: --runonce="false"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801780    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801783    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801786    2567 flags.go:64] FLAG: --seccomp-default="false"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801789    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801792    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801796    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801799    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801802    2567 flags.go:64] FLAG: --storage-driver-password="root"
May 20 15:31:50.806225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801805    2567 flags.go:64] FLAG: --storage-driver-secure="false"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801808    2567 flags.go:64] FLAG: --storage-driver-table="stats"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801810    2567 flags.go:64] FLAG: --storage-driver-user="root"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801813    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801816    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801819    2567 flags.go:64] FLAG: --system-cgroups=""
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801821    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801828    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801831    2567 flags.go:64] FLAG: --tls-cert-file=""
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801834    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801839    2567 flags.go:64] FLAG: --tls-min-version=""
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801842    2567 flags.go:64] FLAG: --tls-private-key-file=""
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801844    2567 flags.go:64] FLAG: --topology-manager-policy="none"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801847    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801850    2567 flags.go:64] FLAG: --topology-manager-scope="container"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801853    2567 flags.go:64] FLAG: --v="2"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801857    2567 flags.go:64] FLAG: --version="false"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801863    2567 flags.go:64] FLAG: --vmodule=""
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801869    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.801872    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.801976    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.801980    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.801983    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.801986    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:50.806840 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.801989    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.801991    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.801994    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.801996    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.801999    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802001    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802005    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802010    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802014    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802017    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802019    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802022    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802025    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802027    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802030    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802032    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802035    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802037    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802039    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:50.807455 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802042    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802044    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802047    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802049    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802051    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802054    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802058    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802062    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802065    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802067    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802070    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802073    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802075    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802078    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802081    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802084    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802086    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802089    2567 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802091    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802094    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:50.807958 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802096    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802098    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802101    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802103    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802105    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802108    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802110    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802113    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802115    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802118    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802120    2567 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802123    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802125    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802127    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802130    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802132    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802134    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802137    2567 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802140    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802144    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:50.808446 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802147    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802149    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802152    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802154    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802157    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802159    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802162    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802165    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802168    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802170    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802173    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802175    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802178    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802180    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802183    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802185    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802187    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802190    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802192    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802195    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:50.808986 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802198    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:50.809502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802200    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:50.809502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.802202    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:50.809502 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.803190    2567 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:50.810880 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.810860    2567 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 20 15:31:50.810923 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.810881    2567 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 20 15:31:50.810953 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810932    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:50.810953 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810940    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:50.810953 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810944    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:50.810953 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810947    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:50.810953 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810949    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:50.810953 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810952    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:50.810953 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810955    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810958    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810961    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810964    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810966    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810969    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810971    2567 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810975    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810978    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810981    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810984    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810986    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810988    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810991    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810993    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810996    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.810999    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811001    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811004    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:50.811131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811006    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811008    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811011    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811014    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811016    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811018    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811021    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811023    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811026    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811028    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811030    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811034    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811036    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811039    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811042    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811045    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811048    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811051    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811053    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811056    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:50.811617 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811058    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811061    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811064    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811066    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811069    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811072    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811074    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811076    2567 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811079    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811081    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811084    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811086    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811089    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811091    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811094    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811096    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811098    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811101    2567 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811103    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811105    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:50.812105 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811108    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811110    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811113    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811115    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811118    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811120    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811122    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811126    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811129    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811131    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811135    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811138    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811141    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811144    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811147    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811150    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811152    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811155    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811157    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811160    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:50.812620 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811162    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.811167    2567 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811262    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811266    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811269    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811272    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811275    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811278    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811281    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811283    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811286    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811288    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811291    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811293    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811295    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811298    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 20 15:31:50.813394 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811300    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811303    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811305    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811308    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811311    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811313    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811316    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811318    2567 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811321    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811323    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811326    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811329    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811331    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811334    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811337    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811339    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811342    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811344    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811346    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 20 15:31:50.813999 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811349    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811351    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811354    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811356    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811359    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811361    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811364    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811366    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811369    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811371    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811373    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811376    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811378    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811381    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811383    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811385    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811388    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811391    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811394    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811396    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 20 15:31:50.814506 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811399    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811403    2567 feature_gate.go:328] unrecognized feature gate: Example
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811405    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811408    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811410    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811413    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811416    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811437    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811441    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811444    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811446    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811449    2567 feature_gate.go:328] unrecognized feature gate: Example2
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811452    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811456    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811459    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811462    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811464    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811467    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811470    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811473    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 20 15:31:50.815013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811476    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811479    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811481    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811484    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811487    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811489    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811492    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811494    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811497    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811500    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811503    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811505    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:50.811508    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.811512    2567 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 20 15:31:50.815502 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.811648    2567 server.go:962] "Client rotation is on, will bootstrap in background"
May 20 15:31:50.816295 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.816273    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 20 15:31:50.817771 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.817758    2567 server.go:1019] "Starting client certificate rotation"
May 20 15:31:50.817889 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.817869    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 15:31:50.819156 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.819141    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 20 15:31:50.860688 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.860655    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 15:31:50.866294 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.866271    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 20 15:31:50.883458 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.883430    2567 log.go:25] "Validated CRI v1 runtime API"
May 20 15:31:50.891360 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.891337    2567 log.go:25] "Validated CRI v1 image API"
May 20 15:31:50.893086 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.893068    2567 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 20 15:31:50.894947 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.894927    2567 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 15:31:50.897679 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.897658    2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a4598b88-07b0-4c01-a066-9f77bac1d1f6:/dev/nvme0n1p4 f2934b5f-a97c-442f-a256-057be2d409c0:/dev/nvme0n1p3]
May 20 15:31:50.897770 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.897678    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 20 15:31:50.902977 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.902553    2567 manager.go:217] Machine: {Timestamp:2026-05-20 15:31:50.901902947 +0000 UTC m=+0.605996528 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110967 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ff219868f7ecf1693bdd04e10db7d SystemUUID:ec2ff219-868f-7ecf-1693-bdd04e10db7d BootID:bf70e3f4-4b94-48cc-b8be-c5c4f8803ae7 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:46:6f:1a:e0:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:46:6f:1a:e0:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:f8:36:93:a4:40 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 20 15:31:50.902977 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.902971    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 20 15:31:50.903113 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.903077    2567 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 20 15:31:50.905549 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.905517    2567 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 20 15:31:50.905705 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.905552    2567 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-227.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 20 15:31:50.905750 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.905718    2567 topology_manager.go:138] "Creating topology manager with none policy"
May 20 15:31:50.905750 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.905727    2567 container_manager_linux.go:306] "Creating device plugin manager"
May 20 15:31:50.905750 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.905735    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 15:31:50.907067 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.907056    2567 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 20 15:31:50.908656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.908643    2567 state_mem.go:36] "Initialized new in-memory state store"
May 20 15:31:50.908955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.908944    2567 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 20 15:31:50.913171 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.913159    2567 kubelet.go:491] "Attempting to sync node with API server"
May 20 15:31:50.913215 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.913175    2567 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 20 15:31:50.913215 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.913188    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 20 15:31:50.913215 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.913201    2567 kubelet.go:397] "Adding apiserver pod source"
May 20 15:31:50.913215 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.913209    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 20 15:31:50.914690 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.914675    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 15:31:50.914742 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.914703    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 20 15:31:50.919315 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.919294    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n5bs8"
May 20 15:31:50.919422 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.919320    2567 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 20 15:31:50.922270 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.922244    2567 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 20 15:31:50.924861 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924845    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 20 15:31:50.924861 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924864    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 20 15:31:50.924982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924870    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 20 15:31:50.924982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924876    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 20 15:31:50.924982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924886    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 20 15:31:50.924982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924896    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 20 15:31:50.924982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924904    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 20 15:31:50.924982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924910    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 20 15:31:50.924982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924917    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 20 15:31:50.924982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924924    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 20 15:31:50.924982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924933    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 20 15:31:50.924982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.924939    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 20 15:31:50.927415 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.927395    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n5bs8"
May 20 15:31:50.927948 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.927934    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 20 15:31:50.927948 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.927950    2567 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 20 15:31:50.928664 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:50.928643    2567 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 20 15:31:50.928664 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:50.928642    2567 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 20 15:31:50.931477 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.931461    2567 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 20 15:31:50.931560 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.931502    2567 server.go:1295] "Started kubelet"
May 20 15:31:50.931652 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.931609    2567 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 20 15:31:50.931711 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.931669    2567 server_v1.go:47] "podresources" method="list" useActivePods=true
May 20 15:31:50.932529 ip-10-0-130-227 systemd[1]: Started Kubernetes Kubelet.
May 20 15:31:50.933067 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.932754    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 20 15:31:50.933552 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.933484    2567 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 20 15:31:50.936485 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.936470    2567 server.go:317] "Adding debug handlers to kubelet server"
May 20 15:31:50.943322 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.943300    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 20 15:31:50.944189 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.944172    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 20 15:31:50.944288 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.944274    2567 volume_manager.go:295] "The desired_state_of_world populator starts"
May 20 15:31:50.944342 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.944289    2567 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 20 15:31:50.944386 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.944376    2567 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 20 15:31:50.944418 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.944413    2567 reconstruct.go:97] "Volume reconstruction finished"
May 20 15:31:50.944418 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.944418    2567 reconciler.go:26] "Reconciler: start to sync state"
May 20 15:31:50.944674 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.944653    2567 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-227.ec2.internal" not found
May 20 15:31:50.945257 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:50.945017    2567 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 20 15:31:50.945257 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.945230    2567 factory.go:55] Registering systemd factory
May 20 15:31:50.945359 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.945289    2567 factory.go:223] Registration of the systemd container factory successfully
May 20 15:31:50.945462 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:50.945443    2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:50.945603 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.945570    2567 factory.go:153] Registering CRI-O factory
May 20 15:31:50.945603 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.945606    2567 factory.go:223] Registration of the crio container factory successfully
May 20 15:31:50.945763 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.945659    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 20 15:31:50.945763 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.945675    2567 factory.go:103] Registering Raw factory
May 20 15:31:50.945763 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.945688    2567 manager.go:1196] Started watching for new ooms in manager
May 20 15:31:50.946500 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.946482    2567 manager.go:319] Starting recovery of all containers
May 20 15:31:50.949084 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.949058    2567 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:50.950454 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:50.950432    2567 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-227.ec2.internal\" not found" node="ip-10-0-130-227.ec2.internal"
May 20 15:31:50.957072 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.956896    2567 manager.go:324] Recovery completed
May 20 15:31:50.960639 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.960617    2567 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-227.ec2.internal" not found
May 20 15:31:50.961527 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.961515    2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:50.963755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.963739    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:50.963843 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.963774    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:50.963843 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.963789    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:50.964242 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.964229    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 20 15:31:50.964304 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.964241    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 20 15:31:50.964304 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.964260    2567 state_mem.go:36] "Initialized new in-memory state store"
May 20 15:31:50.968666 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.968652    2567 policy_none.go:49] "None policy: Start"
May 20 15:31:50.968750 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.968671    2567 memory_manager.go:186] "Starting memorymanager" policy="None"
May 20 15:31:50.968750 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:50.968685    2567 state_mem.go:35] "Initializing new in-memory state store"
May 20 15:31:51.012314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.012294    2567 manager.go:341] "Starting Device Plugin manager"
May 20 15:31:51.026484 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.012326    2567 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 20 15:31:51.026484 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.012336    2567 server.go:85] "Starting device plugin registration server"
May 20 15:31:51.026484 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.012622    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
May 20 15:31:51.026484 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.012633    2567 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 20 15:31:51.026484 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.012711    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 20 15:31:51.026484 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.012789    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 20 15:31:51.026484 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.012796    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 20 15:31:51.026484 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.013338    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 20 15:31:51.026484 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.013373    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:51.026484 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.021649    2567 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-130-227.ec2.internal" not found
May 20 15:31:51.050477 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.050440    2567 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 20 15:31:51.051774 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.051757    2567 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 20 15:31:51.051826 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.051794    2567 status_manager.go:230] "Starting to sync pod status with apiserver"
May 20 15:31:51.051826 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.051821    2567 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 20 15:31:51.051905 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.051830    2567 kubelet.go:2451] "Starting kubelet main sync loop"
May 20 15:31:51.051905 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.051885    2567 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 20 15:31:51.054085 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.054061    2567 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:51.113192 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.113158    2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:51.115211 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.115197    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:51.115268 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.115227    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:51.115268 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.115238    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:51.115268 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.115265    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-227.ec2.internal"
May 20 15:31:51.124862 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.124839    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-227.ec2.internal"
May 20 15:31:51.124934 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.124868    2567 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-227.ec2.internal\": node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:51.145024 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.145004    2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:51.152089 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.152063    2567 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-227.ec2.internal"]
May 20 15:31:51.152149 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.152137    2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:51.154693 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.154678    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:51.154753 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.154704    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:51.154753 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.154714    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:51.155914 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.155901    2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:51.156080 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.156066    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.156125 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.156100    2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:51.156761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.156745    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:51.156823 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.156742    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:51.156823 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.156779    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:51.156823 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.156795    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:51.156823 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.156798    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:51.156823 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.156811    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:51.157975 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.157959    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.158074 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.157985    2567 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 20 15:31:51.159514 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.159496    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientMemory"
May 20 15:31:51.159635 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.159525    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasNoDiskPressure"
May 20 15:31:51.159635 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.159534    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeHasSufficientPID"
May 20 15:31:51.189481 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.189447    2567 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-227.ec2.internal\" not found" node="ip-10-0-130-227.ec2.internal"
May 20 15:31:51.193954 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.193935    2567 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-227.ec2.internal\" not found" node="ip-10-0-130-227.ec2.internal"
May 20 15:31:51.245015 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.244979    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a068ba9a88bc229cfafccce16e49d8ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal\" (UID: \"a068ba9a88bc229cfafccce16e49d8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.245139 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.245048    2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:51.345598 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.345505    2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:51.345693 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.345625    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a068ba9a88bc229cfafccce16e49d8ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal\" (UID: \"a068ba9a88bc229cfafccce16e49d8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.345693 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.345662    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a068ba9a88bc229cfafccce16e49d8ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal\" (UID: \"a068ba9a88bc229cfafccce16e49d8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.345693 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.345680    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a90538a519c49afc5a39f91384815237-config\") pod \"kube-apiserver-proxy-ip-10-0-130-227.ec2.internal\" (UID: \"a90538a519c49afc5a39f91384815237\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.345784 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.345718    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a068ba9a88bc229cfafccce16e49d8ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal\" (UID: \"a068ba9a88bc229cfafccce16e49d8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.446175 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.446129    2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:51.446175 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.446146    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a068ba9a88bc229cfafccce16e49d8ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal\" (UID: \"a068ba9a88bc229cfafccce16e49d8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.446278 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.446182    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a068ba9a88bc229cfafccce16e49d8ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal\" (UID: \"a068ba9a88bc229cfafccce16e49d8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.446278 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.446214    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a90538a519c49afc5a39f91384815237-config\") pod \"kube-apiserver-proxy-ip-10-0-130-227.ec2.internal\" (UID: \"a90538a519c49afc5a39f91384815237\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.446278 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.446240    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a90538a519c49afc5a39f91384815237-config\") pod \"kube-apiserver-proxy-ip-10-0-130-227.ec2.internal\" (UID: \"a90538a519c49afc5a39f91384815237\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.492340 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.492301    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.495875 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.495855    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-227.ec2.internal"
May 20 15:31:51.546786 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.546738    2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:51.647174 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.647090    2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:51.747549 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.747517    2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:51.816686 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.816653    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 20 15:31:51.817239 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.816828    2567 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 15:31:51.817239 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.816848    2567 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 20 15:31:51.848000 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.847971    2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:51.930689 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.930599    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-19 15:26:50 +0000 UTC" deadline="2027-10-18 12:59:59.511163954 +0000 UTC"
May 20 15:31:51.930689 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.930643    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12381h28m7.580525095s"
May 20 15:31:51.943754 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.943727    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 20 15:31:51.948297 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:51.948273    2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:51.965915 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:51.965870    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90538a519c49afc5a39f91384815237.slice/crio-cf938bc58c5db828aa4766b7f0500cd98b470730535695b2ef2dfe8e4e3b1785 WatchSource:0}: Error finding container cf938bc58c5db828aa4766b7f0500cd98b470730535695b2ef2dfe8e4e3b1785: Status 404 returned error can't find the container with id cf938bc58c5db828aa4766b7f0500cd98b470730535695b2ef2dfe8e4e3b1785
May 20 15:31:51.966111 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:51.966086    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda068ba9a88bc229cfafccce16e49d8ae.slice/crio-a919645a0cd6e639eadd12c476f175e8cd7333bb250ff7c496f07ccd66f7b555 WatchSource:0}: Error finding container a919645a0cd6e639eadd12c476f175e8cd7333bb250ff7c496f07ccd66f7b555: Status 404 returned error can't find the container with id a919645a0cd6e639eadd12c476f175e8cd7333bb250ff7c496f07ccd66f7b555
May 20 15:31:51.970503 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.969267    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:31:51.971522 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:51.971502    2567 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 20 15:31:52.048899 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:52.048862    2567 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-130-227.ec2.internal\" not found"
May 20 15:31:52.054716 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.054668    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal" event={"ID":"a068ba9a88bc229cfafccce16e49d8ae","Type":"ContainerStarted","Data":"a919645a0cd6e639eadd12c476f175e8cd7333bb250ff7c496f07ccd66f7b555"}
May 20 15:31:52.055623 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.055603    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-227.ec2.internal" event={"ID":"a90538a519c49afc5a39f91384815237","Type":"ContainerStarted","Data":"cf938bc58c5db828aa4766b7f0500cd98b470730535695b2ef2dfe8e4e3b1785"}
May 20 15:31:52.075648 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.075620    2567 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:52.102705 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.102670    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cf7c8"
May 20 15:31:52.137741 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.137716    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cf7c8"
May 20 15:31:52.144853 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.144828    2567 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal"
May 20 15:31:52.213386 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.213298    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 15:31:52.215051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.215031    2567 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-227.ec2.internal"
May 20 15:31:52.239002 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.238974    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 20 15:31:52.361960 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.361835    2567 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:52.778051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.778017    2567 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:52.914393 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.914361    2567 apiserver.go:52] "Watching apiserver"
May 20 15:31:52.920710 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.920686    2567 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 20 15:31:52.922640 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.922610    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-b25vj","kube-system/kube-apiserver-proxy-ip-10-0-130-227.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5","openshift-cluster-node-tuning-operator/tuned-mpmsj","openshift-image-registry/node-ca-ccd2v","openshift-multus/multus-4t7b8","openshift-ovn-kubernetes/ovnkube-node-dbwqx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal","openshift-multus/multus-additional-cni-plugins-fsxb5","openshift-multus/network-metrics-daemon-46dsm","openshift-network-diagnostics/network-check-target-q4gpd","openshift-network-operator/iptables-alerter-sb4fl"]
May 20 15:31:52.926466 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.926205    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b25vj"
May 20 15:31:52.927457 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.927438    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:52.928834 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.928814    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.928944 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.928893    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ccd2v"
May 20 15:31:52.929436 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.929415    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-55f6w\""
May 20 15:31:52.929538 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.929467    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 20 15:31:52.930141 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.929944    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 20 15:31:52.930141 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.930018    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 20 15:31:52.930141 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.930030    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 20 15:31:52.930342 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.930238    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pt4bm\""
May 20 15:31:52.930601 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.930571    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 20 15:31:52.930854 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.930838    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.931447 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.931429    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ttprt\""
May 20 15:31:52.931540 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.931497    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rjzrw\""
May 20 15:31:52.931629 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.931616    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 20 15:31:52.931734 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.931428    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 20 15:31:52.931954 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.931940    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 20 15:31:52.932099 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.932083    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 20 15:31:52.932180 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.932111    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 20 15:31:52.932408 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.932391    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.933141 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.933115    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 20 15:31:52.933477 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.933458    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 20 15:31:52.934266 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.933816    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xtdrr\""
May 20 15:31:52.934266 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.934061    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 20 15:31:52.934266 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.934097    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 20 15:31:52.934987 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.934969    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 20 15:31:52.935076 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.935028    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-764gr\""
May 20 15:31:52.936235 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.935374    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 20 15:31:52.936235 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.935385    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 20 15:31:52.936235 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.935452    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:52.936235 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.935378    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 20 15:31:52.936235 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.935833    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 20 15:31:52.936235 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.935854    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 20 15:31:52.936794 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.936768    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:52.936884 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:52.936839    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:31:52.938273 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.937983    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 20 15:31:52.938273 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.937988    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 20 15:31:52.938273 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.938082    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zx8cc\""
May 20 15:31:52.938429 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.938404    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:31:52.938491 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:52.938472    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:31:52.938549 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.938405    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sb4fl"
May 20 15:31:52.942341 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.941442    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 20 15:31:52.944869 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.942975    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j66cd\""
May 20 15:31:52.944869 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.943043    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 20 15:31:52.944869 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.943503    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 20 15:31:52.946820 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.946801    2567 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 20 15:31:52.953249 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953227    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:52.953344 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953265    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7c4\" (UniqueName: \"kubernetes.io/projected/d6731423-938b-453d-965d-db728cfa0efb-kube-api-access-zs7c4\") pod \"iptables-alerter-sb4fl\" (UID: \"d6731423-938b-453d-965d-db728cfa0efb\") " pod="openshift-network-operator/iptables-alerter-sb4fl"
May 20 15:31:52.953344 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953291    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-host\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.953426 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/71f15969-7e79-4580-b24d-41f479c3715e-agent-certs\") pod \"konnectivity-agent-b25vj\" (UID: \"71f15969-7e79-4580-b24d-41f479c3715e\") " pod="kube-system/konnectivity-agent-b25vj"
May 20 15:31:52.953426 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953396    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-etc-kubernetes\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.953426 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953415    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-slash\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.953532 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953429    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:52.953532 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953445    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6731423-938b-453d-965d-db728cfa0efb-host-slash\") pod \"iptables-alerter-sb4fl\" (UID: \"d6731423-938b-453d-965d-db728cfa0efb\") " pod="openshift-network-operator/iptables-alerter-sb4fl"
May 20 15:31:52.953532 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953472    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/581482dd-fca5-4834-99f4-519a6df326fb-ovnkube-config\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.953532 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953493    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/581482dd-fca5-4834-99f4-519a6df326fb-ovn-node-metrics-cert\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.953532 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953518    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-modprobe-d\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.953758 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953539    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65cd891-4c27-4070-972c-9a97f6fc62af-tmp\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.953758 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953563    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e367b95c-7b4d-4d9c-8a71-1429e26b0230-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:52.953758 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953615    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-var-lib-cni-bin\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.953758 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953651    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-run-openvswitch\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.953758 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953671    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffa2ed51-616b-4441-a33e-45c761dd075e-host\") pod \"node-ca-ccd2v\" (UID: \"ffa2ed51-616b-4441-a33e-45c761dd075e\") " pod="openshift-image-registry/node-ca-ccd2v"
May 20 15:31:52.953758 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953715    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-cnibin\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:52.954022 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953772    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-cni-dir\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.954022 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953805    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-hostroot\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.954022 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953837    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-systemd-units\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.954022 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953876    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6qd\" (UniqueName: \"kubernetes.io/projected/04986e91-b72e-47f8-a17f-c6390009d60b-kube-api-access-nb6qd\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:52.954022 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953936    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-var-lib-kubelet\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.954022 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.953975    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-daemon-config\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.954022 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954002    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.954330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954034    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-system-cni-dir\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:52.954330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954058    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-682mt\" (UniqueName: \"kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt\") pod \"network-check-target-q4gpd\" (UID: \"ddb4cb56-7904-4742-ac26-ecbd4f38df78\") " pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:31:52.954330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954080    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:52.954330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktq2w\" (UniqueName: \"kubernetes.io/projected/f65cd891-4c27-4070-972c-9a97f6fc62af-kube-api-access-ktq2w\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.954330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954127    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffa2ed51-616b-4441-a33e-45c761dd075e-serviceca\") pod \"node-ca-ccd2v\" (UID: \"ffa2ed51-616b-4441-a33e-45c761dd075e\") " pod="openshift-image-registry/node-ca-ccd2v"
May 20 15:31:52.954330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954167    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssz7p\" (UniqueName: \"kubernetes.io/projected/ffa2ed51-616b-4441-a33e-45c761dd075e-kube-api-access-ssz7p\") pod \"node-ca-ccd2v\" (UID: \"ffa2ed51-616b-4441-a33e-45c761dd075e\") " pod="openshift-image-registry/node-ca-ccd2v"
May 20 15:31:52.954330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954197    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e367b95c-7b4d-4d9c-8a71-1429e26b0230-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:52.954330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954222    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-run-netns\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.954330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954246    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-system-cni-dir\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.954330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954268    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-run-ovn\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.954330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954302    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-sysconfig\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954343    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-os-release\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954391    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-var-lib-cni-multus\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954438    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-run-multus-certs\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954466    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-systemd\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954490    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-lib-modules\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954530    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-socket-dir-parent\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954557    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-run-k8s-cni-cncf-io\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954600    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/71f15969-7e79-4580-b24d-41f479c3715e-konnectivity-ca\") pod \"konnectivity-agent-b25vj\" (UID: \"71f15969-7e79-4580-b24d-41f479c3715e\") " pod="kube-system/konnectivity-agent-b25vj"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954625    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-run-systemd\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954641    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-device-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954662    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-sys\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954693    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/581482dd-fca5-4834-99f4-519a6df326fb-env-overrides\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954715    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-kubernetes\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.954733 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954734    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-run\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954763    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkn24\" (UniqueName: \"kubernetes.io/projected/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-kube-api-access-tkn24\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954790    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-conf-dir\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954811    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchzp\" (UniqueName: \"kubernetes.io/projected/6b36f593-5d2c-45dc-b0ff-bd5e90381257-kube-api-access-hchzp\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954832    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-kubelet\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954855    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-node-log\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954880    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-cni-bin\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954902    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-cni-netd\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954923    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-var-lib-kubelet\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954938    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-tuned\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954952    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b36f593-5d2c-45dc-b0ff-bd5e90381257-cni-binary-copy\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.954982    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-var-lib-openvswitch\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955007    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-etc-openvswitch\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955021    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-log-socket\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955041    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955062    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsw4g\" (UniqueName: \"kubernetes.io/projected/581482dd-fca5-4834-99f4-519a6df326fb-kube-api-access-vsw4g\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.955271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955100    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-sys-fs\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955153    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-sysctl-conf\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955177    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-cnibin\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955195    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-os-release\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955217    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-socket-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955269    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-sysctl-d\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955312    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e367b95c-7b4d-4d9c-8a71-1429e26b0230-cni-binary-copy\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955346    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnzzb\" (UniqueName: \"kubernetes.io/projected/e367b95c-7b4d-4d9c-8a71-1429e26b0230-kube-api-access-hnzzb\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955370    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d6731423-938b-453d-965d-db728cfa0efb-iptables-alerter-script\") pod \"iptables-alerter-sb4fl\" (UID: \"d6731423-938b-453d-965d-db728cfa0efb\") " pod="openshift-network-operator/iptables-alerter-sb4fl"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955437    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-run-netns\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955463    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/581482dd-fca5-4834-99f4-519a6df326fb-ovnkube-script-lib\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955487    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-registration-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:52.955883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:52.955515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.056235 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056154    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e367b95c-7b4d-4d9c-8a71-1429e26b0230-cni-binary-copy\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.056235 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056198    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hnzzb\" (UniqueName: \"kubernetes.io/projected/e367b95c-7b4d-4d9c-8a71-1429e26b0230-kube-api-access-hnzzb\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.056235 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056226    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d6731423-938b-453d-965d-db728cfa0efb-iptables-alerter-script\") pod \"iptables-alerter-sb4fl\" (UID: \"d6731423-938b-453d-965d-db728cfa0efb\") " pod="openshift-network-operator/iptables-alerter-sb4fl"
May 20 15:31:53.056444 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056252    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-run-netns\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.056444 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056272    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/581482dd-fca5-4834-99f4-519a6df326fb-ovnkube-script-lib\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.056444 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056294    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-registration-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.056444 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056310    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.056444 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056324    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.056444 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056341    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7c4\" (UniqueName: \"kubernetes.io/projected/d6731423-938b-453d-965d-db728cfa0efb-kube-api-access-zs7c4\") pod \"iptables-alerter-sb4fl\" (UID: \"d6731423-938b-453d-965d-db728cfa0efb\") " pod="openshift-network-operator/iptables-alerter-sb4fl"
May 20 15:31:53.056444 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056357    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-host\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.056444 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056388    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/71f15969-7e79-4580-b24d-41f479c3715e-agent-certs\") pod \"konnectivity-agent-b25vj\" (UID: \"71f15969-7e79-4580-b24d-41f479c3715e\") " pod="kube-system/konnectivity-agent-b25vj"
May 20 15:31:53.056444 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056413    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-etc-kubernetes\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056466    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-etc-kubernetes\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056489    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-registration-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056526    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056534    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-slash\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056565    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056617    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6731423-938b-453d-965d-db728cfa0efb-host-slash\") pod \"iptables-alerter-sb4fl\" (UID: \"d6731423-938b-453d-965d-db728cfa0efb\") " pod="openshift-network-operator/iptables-alerter-sb4fl"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056662    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/581482dd-fca5-4834-99f4-519a6df326fb-ovnkube-config\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056673    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-host\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056691    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056700    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/581482dd-fca5-4834-99f4-519a6df326fb-ovn-node-metrics-cert\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056728    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-modprobe-d\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056757    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65cd891-4c27-4070-972c-9a97f6fc62af-tmp\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.056786 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056786    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e367b95c-7b4d-4d9c-8a71-1429e26b0230-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056806    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e367b95c-7b4d-4d9c-8a71-1429e26b0230-cni-binary-copy\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056854    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-var-lib-cni-bin\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056881    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-run-openvswitch\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056903    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffa2ed51-616b-4441-a33e-45c761dd075e-host\") pod \"node-ca-ccd2v\" (UID: \"ffa2ed51-616b-4441-a33e-45c761dd075e\") " pod="openshift-image-registry/node-ca-ccd2v"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.056926    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056986    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-cnibin\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057054    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-slash\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.057109    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs podName:9f8d9d11-6bdb-402e-ab74-7485d6c52a79 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:53.557055212 +0000 UTC m=+3.261148800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs") pod "network-metrics-daemon-46dsm" (UID: "9f8d9d11-6bdb-402e-ab74-7485d6c52a79") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.056927    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-cnibin\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057150    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-cni-dir\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057185    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-hostroot\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057210    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-systemd-units\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057222    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/581482dd-fca5-4834-99f4-519a6df326fb-ovnkube-script-lib\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057227    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057235    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6qd\" (UniqueName: \"kubernetes.io/projected/04986e91-b72e-47f8-a17f-c6390009d60b-kube-api-access-nb6qd\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057264    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-var-lib-kubelet\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057261    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-run-netns\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.057365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057296    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-daemon-config\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057300    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d6731423-938b-453d-965d-db728cfa0efb-iptables-alerter-script\") pod \"iptables-alerter-sb4fl\" (UID: \"d6731423-938b-453d-965d-db728cfa0efb\") " pod="openshift-network-operator/iptables-alerter-sb4fl"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057323    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057353    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-system-cni-dir\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057360    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-var-lib-cni-bin\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057391    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-682mt\" (UniqueName: \"kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt\") pod \"network-check-target-q4gpd\" (UID: \"ddb4cb56-7904-4742-ac26-ecbd4f38df78\") " pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057408    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-system-cni-dir\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057420    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057436    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-modprobe-d\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057451    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ktq2w\" (UniqueName: \"kubernetes.io/projected/f65cd891-4c27-4070-972c-9a97f6fc62af-kube-api-access-ktq2w\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057476    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffa2ed51-616b-4441-a33e-45c761dd075e-serviceca\") pod \"node-ca-ccd2v\" (UID: \"ffa2ed51-616b-4441-a33e-45c761dd075e\") " pod="openshift-image-registry/node-ca-ccd2v"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057500    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ssz7p\" (UniqueName: \"kubernetes.io/projected/ffa2ed51-616b-4441-a33e-45c761dd075e-kube-api-access-ssz7p\") pod \"node-ca-ccd2v\" (UID: \"ffa2ed51-616b-4441-a33e-45c761dd075e\") " pod="openshift-image-registry/node-ca-ccd2v"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057518    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/581482dd-fca5-4834-99f4-519a6df326fb-ovnkube-config\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057525    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e367b95c-7b4d-4d9c-8a71-1429e26b0230-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057537    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-var-lib-kubelet\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057561    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-run-netns\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057326    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-run-openvswitch\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.058219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057597    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-hostroot\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057635    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-run-netns\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057637    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-cni-dir\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057669    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-systemd-units\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057699    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057732    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-system-cni-dir\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057762    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-run-ovn\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057805    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-run-ovn\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057815    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffa2ed51-616b-4441-a33e-45c761dd075e-host\") pod \"node-ca-ccd2v\" (UID: \"ffa2ed51-616b-4441-a33e-45c761dd075e\") " pod="openshift-image-registry/node-ca-ccd2v"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057849    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.057939    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-system-cni-dir\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058022    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffa2ed51-616b-4441-a33e-45c761dd075e-serviceca\") pod \"node-ca-ccd2v\" (UID: \"ffa2ed51-616b-4441-a33e-45c761dd075e\") " pod="openshift-image-registry/node-ca-ccd2v"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058080    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-sysconfig\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058110    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-os-release\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058134    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-var-lib-cni-multus\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058161    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-run-multus-certs\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058186    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-systemd\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058209    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-lib-modules\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.059132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058248    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e367b95c-7b4d-4d9c-8a71-1429e26b0230-os-release\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058251    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-daemon-config\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058252    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-socket-dir-parent\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058299    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-run-k8s-cni-cncf-io\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058309    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-run-multus-certs\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058310    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-socket-dir-parent\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058326    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/71f15969-7e79-4580-b24d-41f479c3715e-konnectivity-ca\") pod \"konnectivity-agent-b25vj\" (UID: \"71f15969-7e79-4580-b24d-41f479c3715e\") " pod="kube-system/konnectivity-agent-b25vj"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058342    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-sysconfig\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058352    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-run-systemd\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058363    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-var-lib-cni-multus\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058379    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-device-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058390    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-systemd\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058403    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-sys\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058424    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e367b95c-7b4d-4d9c-8a71-1429e26b0230-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058428    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/581482dd-fca5-4834-99f4-519a6df326fb-env-overrides\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058469    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-kubernetes\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058487    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-lib-modules\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.059962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058498    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-run\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058528    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tkn24\" (UniqueName: \"kubernetes.io/projected/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-kube-api-access-tkn24\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058535    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-kubernetes\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058553    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-conf-dir\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058575    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d6731423-938b-453d-965d-db728cfa0efb-host-slash\") pod \"iptables-alerter-sb4fl\" (UID: \"d6731423-938b-453d-965d-db728cfa0efb\") " pod="openshift-network-operator/iptables-alerter-sb4fl"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058578    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hchzp\" (UniqueName: \"kubernetes.io/projected/6b36f593-5d2c-45dc-b0ff-bd5e90381257-kube-api-access-hchzp\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058631    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-kubelet\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058657    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-node-log\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058680    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-cni-bin\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058705    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-cni-netd\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058729    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-var-lib-kubelet\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058753    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-tuned\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058776    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b36f593-5d2c-45dc-b0ff-bd5e90381257-cni-binary-copy\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058799    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/581482dd-fca5-4834-99f4-519a6df326fb-env-overrides\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058804    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-var-lib-openvswitch\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058843    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-var-lib-openvswitch\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058856    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-etc-openvswitch\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058880    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-kubelet\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.060709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058885    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-log-socket\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058901    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/71f15969-7e79-4580-b24d-41f479c3715e-konnectivity-ca\") pod \"konnectivity-agent-b25vj\" (UID: \"71f15969-7e79-4580-b24d-41f479c3715e\") " pod="kube-system/konnectivity-agent-b25vj"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058912    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058917    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-node-log\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058940    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw4g\" (UniqueName: \"kubernetes.io/projected/581482dd-fca5-4834-99f4-519a6df326fb-kube-api-access-vsw4g\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058947    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-host-run-k8s-cni-cncf-io\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058955    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-cni-bin\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058966    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-sys-fs\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058990    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-etc-openvswitch\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.058991    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-sysctl-conf\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059004    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-run-systemd\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059006    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-run-ovn-kubernetes\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059029    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-cnibin\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059045    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-sys\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059052    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-os-release\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059078    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-socket-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059091    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-device-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059092    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-sysctl-conf\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.061435 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059107    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-sysctl-d\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059138    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-sys-fs\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059142    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-run\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059176    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-multus-conf-dir\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059188    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-cnibin\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059191    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-var-lib-kubelet\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059235    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b36f593-5d2c-45dc-b0ff-bd5e90381257-os-release\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059285    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/04986e91-b72e-47f8-a17f-c6390009d60b-socket-dir\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059328    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-host-cni-netd\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059368    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/581482dd-fca5-4834-99f4-519a6df326fb-log-socket\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059455    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-sysctl-d\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059576    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b36f593-5d2c-45dc-b0ff-bd5e90381257-cni-binary-copy\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.059948    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e367b95c-7b4d-4d9c-8a71-1429e26b0230-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.060852    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65cd891-4c27-4070-972c-9a97f6fc62af-tmp\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.060954    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/581482dd-fca5-4834-99f4-519a6df326fb-ovn-node-metrics-cert\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.061723    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f65cd891-4c27-4070-972c-9a97f6fc62af-etc-tuned\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.062165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.061993    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/71f15969-7e79-4580-b24d-41f479c3715e-agent-certs\") pod \"konnectivity-agent-b25vj\" (UID: \"71f15969-7e79-4580-b24d-41f479c3715e\") " pod="kube-system/konnectivity-agent-b25vj"
May 20 15:31:53.089109 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.089079    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7c4\" (UniqueName: \"kubernetes.io/projected/d6731423-938b-453d-965d-db728cfa0efb-kube-api-access-zs7c4\") pod \"iptables-alerter-sb4fl\" (UID: \"d6731423-938b-453d-965d-db728cfa0efb\") " pod="openshift-network-operator/iptables-alerter-sb4fl"
May 20 15:31:53.103775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.103743    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsw4g\" (UniqueName: \"kubernetes.io/projected/581482dd-fca5-4834-99f4-519a6df326fb-kube-api-access-vsw4g\") pod \"ovnkube-node-dbwqx\" (UID: \"581482dd-fca5-4834-99f4-519a6df326fb\") " pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.104009 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.103981    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktq2w\" (UniqueName: \"kubernetes.io/projected/f65cd891-4c27-4070-972c-9a97f6fc62af-kube-api-access-ktq2w\") pod \"tuned-mpmsj\" (UID: \"f65cd891-4c27-4070-972c-9a97f6fc62af\") " pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.104115 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.104083    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:53.104115 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.104103    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:53.104231 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.104118    2567 projected.go:194] Error preparing data for projected volume kube-api-access-682mt for pod openshift-network-diagnostics/network-check-target-q4gpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:53.104231 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.104134    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnzzb\" (UniqueName: \"kubernetes.io/projected/e367b95c-7b4d-4d9c-8a71-1429e26b0230-kube-api-access-hnzzb\") pod \"multus-additional-cni-plugins-fsxb5\" (UID: \"e367b95c-7b4d-4d9c-8a71-1429e26b0230\") " pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.104231 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.104185    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt podName:ddb4cb56-7904-4742-ac26-ecbd4f38df78 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:53.604167933 +0000 UTC m=+3.308261504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-682mt" (UniqueName: "kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt") pod "network-check-target-q4gpd" (UID: "ddb4cb56-7904-4742-ac26-ecbd4f38df78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:53.104393 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.104351    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssz7p\" (UniqueName: \"kubernetes.io/projected/ffa2ed51-616b-4441-a33e-45c761dd075e-kube-api-access-ssz7p\") pod \"node-ca-ccd2v\" (UID: \"ffa2ed51-616b-4441-a33e-45c761dd075e\") " pod="openshift-image-registry/node-ca-ccd2v"
May 20 15:31:53.104948 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.104916    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchzp\" (UniqueName: \"kubernetes.io/projected/6b36f593-5d2c-45dc-b0ff-bd5e90381257-kube-api-access-hchzp\") pod \"multus-4t7b8\" (UID: \"6b36f593-5d2c-45dc-b0ff-bd5e90381257\") " pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.105868 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.105844    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6qd\" (UniqueName: \"kubernetes.io/projected/04986e91-b72e-47f8-a17f-c6390009d60b-kube-api-access-nb6qd\") pod \"aws-ebs-csi-driver-node-zwss5\" (UID: \"04986e91-b72e-47f8-a17f-c6390009d60b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.111035 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.111011    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkn24\" (UniqueName: \"kubernetes.io/projected/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-kube-api-access-tkn24\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:53.138930 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.138884    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 15:26:52 +0000 UTC" deadline="2027-10-18 06:04:14.514236846 +0000 UTC"
May 20 15:31:53.138930 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.138931    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12374h32m21.375308883s"
May 20 15:31:53.238730 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.238700    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b25vj"
May 20 15:31:53.245863 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.245838    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5"
May 20 15:31:53.256452 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.256430    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mpmsj"
May 20 15:31:53.260108 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.260089    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ccd2v"
May 20 15:31:53.266719 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.266696    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4t7b8"
May 20 15:31:53.274659 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.274637    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:31:53.277229 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.277208    2567 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 20 15:31:53.279670 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.279652    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fsxb5"
May 20 15:31:53.286211 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.286196    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sb4fl"
May 20 15:31:53.526701 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:53.526566    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581482dd_fca5_4834_99f4_519a6df326fb.slice/crio-ff495d003b0fc47d42e49d8fa9e02585bb5c3fe2f4707eafcf53ae86a41e24d1 WatchSource:0}: Error finding container ff495d003b0fc47d42e49d8fa9e02585bb5c3fe2f4707eafcf53ae86a41e24d1: Status 404 returned error can't find the container with id ff495d003b0fc47d42e49d8fa9e02585bb5c3fe2f4707eafcf53ae86a41e24d1
May 20 15:31:53.530072 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:53.530047    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode367b95c_7b4d_4d9c_8a71_1429e26b0230.slice/crio-488ae7e8ed705293462df24ff592cdb1eafe5f689b0eee7fbeb9bb6c7ae32c93 WatchSource:0}: Error finding container 488ae7e8ed705293462df24ff592cdb1eafe5f689b0eee7fbeb9bb6c7ae32c93: Status 404 returned error can't find the container with id 488ae7e8ed705293462df24ff592cdb1eafe5f689b0eee7fbeb9bb6c7ae32c93
May 20 15:31:53.531562 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:53.531542    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f15969_7e79_4580_b24d_41f479c3715e.slice/crio-52881acf823d103e53bcf29cd62094cd76f3fa3cf0d64273554b13727fc807ee WatchSource:0}: Error finding container 52881acf823d103e53bcf29cd62094cd76f3fa3cf0d64273554b13727fc807ee: Status 404 returned error can't find the container with id 52881acf823d103e53bcf29cd62094cd76f3fa3cf0d64273554b13727fc807ee
May 20 15:31:53.533565 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:53.533543    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04986e91_b72e_47f8_a17f_c6390009d60b.slice/crio-2a8c2108bb85f3069178fa3fa608f9d81935c2182bf9a543f6c303c50d10db87 WatchSource:0}: Error finding container 2a8c2108bb85f3069178fa3fa608f9d81935c2182bf9a543f6c303c50d10db87: Status 404 returned error can't find the container with id 2a8c2108bb85f3069178fa3fa608f9d81935c2182bf9a543f6c303c50d10db87
May 20 15:31:53.535137 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:53.535025    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b36f593_5d2c_45dc_b0ff_bd5e90381257.slice/crio-aaa9018c4f4540a0d62f13842c3be7e345ea6c70ed65fb07af6eeff9f9a27e66 WatchSource:0}: Error finding container aaa9018c4f4540a0d62f13842c3be7e345ea6c70ed65fb07af6eeff9f9a27e66: Status 404 returned error can't find the container with id aaa9018c4f4540a0d62f13842c3be7e345ea6c70ed65fb07af6eeff9f9a27e66
May 20 15:31:53.535979 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:53.535948    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65cd891_4c27_4070_972c_9a97f6fc62af.slice/crio-8ceb4e1d36b845d5a515015ad3b1c3c2f9fa68c30e2123b374162959f25f66fd WatchSource:0}: Error finding container 8ceb4e1d36b845d5a515015ad3b1c3c2f9fa68c30e2123b374162959f25f66fd: Status 404 returned error can't find the container with id 8ceb4e1d36b845d5a515015ad3b1c3c2f9fa68c30e2123b374162959f25f66fd
May 20 15:31:53.538281 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:53.537685    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa2ed51_616b_4441_a33e_45c761dd075e.slice/crio-51d4f14e76e1a91392ede223b957cd0531309f8c8273830586eca530ad532b2f WatchSource:0}: Error finding container 51d4f14e76e1a91392ede223b957cd0531309f8c8273830586eca530ad532b2f: Status 404 returned error can't find the container with id 51d4f14e76e1a91392ede223b957cd0531309f8c8273830586eca530ad532b2f
May 20 15:31:53.539510 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:31:53.538460    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6731423_938b_453d_965d_db728cfa0efb.slice/crio-81af38153751b610199f36a17cc36438a622e72c5c081a94e13b9e74e2dc1df7 WatchSource:0}: Error finding container 81af38153751b610199f36a17cc36438a622e72c5c081a94e13b9e74e2dc1df7: Status 404 returned error can't find the container with id 81af38153751b610199f36a17cc36438a622e72c5c081a94e13b9e74e2dc1df7
May 20 15:31:53.563103 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.562932    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:53.563186 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.563105    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:53.563186 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.563163    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs podName:9f8d9d11-6bdb-402e-ab74-7485d6c52a79 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:54.563146369 +0000 UTC m=+4.267239943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs") pod "network-metrics-daemon-46dsm" (UID: "9f8d9d11-6bdb-402e-ab74-7485d6c52a79") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:53.663866 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:53.663828    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-682mt\" (UniqueName: \"kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt\") pod \"network-check-target-q4gpd\" (UID: \"ddb4cb56-7904-4742-ac26-ecbd4f38df78\") " pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:31:53.664029 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.663971    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:53.664029 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.663997    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:53.664029 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.664009    2567 projected.go:194] Error preparing data for projected volume kube-api-access-682mt for pod openshift-network-diagnostics/network-check-target-q4gpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:53.664132 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:53.664062    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt podName:ddb4cb56-7904-4742-ac26-ecbd4f38df78 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:54.66404782 +0000 UTC m=+4.368141394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-682mt" (UniqueName: "kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt") pod "network-check-target-q4gpd" (UID: "ddb4cb56-7904-4742-ac26-ecbd4f38df78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:54.079939 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.079850    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mpmsj" event={"ID":"f65cd891-4c27-4070-972c-9a97f6fc62af","Type":"ContainerStarted","Data":"8ceb4e1d36b845d5a515015ad3b1c3c2f9fa68c30e2123b374162959f25f66fd"}
May 20 15:31:54.096713 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.096663    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4t7b8" event={"ID":"6b36f593-5d2c-45dc-b0ff-bd5e90381257","Type":"ContainerStarted","Data":"aaa9018c4f4540a0d62f13842c3be7e345ea6c70ed65fb07af6eeff9f9a27e66"}
May 20 15:31:54.110132 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.110047    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx" event={"ID":"581482dd-fca5-4834-99f4-519a6df326fb","Type":"ContainerStarted","Data":"ff495d003b0fc47d42e49d8fa9e02585bb5c3fe2f4707eafcf53ae86a41e24d1"}
May 20 15:31:54.120053 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.120001    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5" event={"ID":"04986e91-b72e-47f8-a17f-c6390009d60b","Type":"ContainerStarted","Data":"2a8c2108bb85f3069178fa3fa608f9d81935c2182bf9a543f6c303c50d10db87"}
May 20 15:31:54.131763 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.131706    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b25vj" event={"ID":"71f15969-7e79-4580-b24d-41f479c3715e","Type":"ContainerStarted","Data":"52881acf823d103e53bcf29cd62094cd76f3fa3cf0d64273554b13727fc807ee"}
May 20 15:31:54.139551 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.139497    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-19 15:26:52 +0000 UTC" deadline="2027-11-23 11:53:52.395611499 +0000 UTC"
May 20 15:31:54.139551 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.139525    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13244h21m58.256090205s"
May 20 15:31:54.151223 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.151165    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsxb5" event={"ID":"e367b95c-7b4d-4d9c-8a71-1429e26b0230","Type":"ContainerStarted","Data":"488ae7e8ed705293462df24ff592cdb1eafe5f689b0eee7fbeb9bb6c7ae32c93"}
May 20 15:31:54.174320 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.174277    2567 generic.go:358] "Generic (PLEG): container finished" podID="a068ba9a88bc229cfafccce16e49d8ae" containerID="5b87558468966eaae8dab80ee8ba5234797e3b59328b641b660c46db59abcc34" exitCode=0
May 20 15:31:54.174481 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.174427    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal" event={"ID":"a068ba9a88bc229cfafccce16e49d8ae","Type":"ContainerDied","Data":"5b87558468966eaae8dab80ee8ba5234797e3b59328b641b660c46db59abcc34"}
May 20 15:31:54.195277 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.195206    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-227.ec2.internal" event={"ID":"a90538a519c49afc5a39f91384815237","Type":"ContainerStarted","Data":"987196c9aa5946163cde2cd2675f232a0870755d6d2101d8b6bff470d3301f25"}
May 20 15:31:54.207059 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.207007    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ccd2v" event={"ID":"ffa2ed51-616b-4441-a33e-45c761dd075e","Type":"ContainerStarted","Data":"51d4f14e76e1a91392ede223b957cd0531309f8c8273830586eca530ad532b2f"}
May 20 15:31:54.209929 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.209882    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sb4fl" event={"ID":"d6731423-938b-453d-965d-db728cfa0efb","Type":"ContainerStarted","Data":"81af38153751b610199f36a17cc36438a622e72c5c081a94e13b9e74e2dc1df7"}
May 20 15:31:54.571574 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.571533    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:54.571764 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:54.571704    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:54.571848 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:54.571772    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs podName:9f8d9d11-6bdb-402e-ab74-7485d6c52a79 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:56.571752433 +0000 UTC m=+6.275846002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs") pod "network-metrics-daemon-46dsm" (UID: "9f8d9d11-6bdb-402e-ab74-7485d6c52a79") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:54.673139 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:54.672464    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-682mt\" (UniqueName: \"kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt\") pod \"network-check-target-q4gpd\" (UID: \"ddb4cb56-7904-4742-ac26-ecbd4f38df78\") " pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:31:54.673139 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:54.672662    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:54.673139 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:54.672685    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:54.673139 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:54.672698    2567 projected.go:194] Error preparing data for projected volume kube-api-access-682mt for pod openshift-network-diagnostics/network-check-target-q4gpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:54.673139 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:54.672756    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt podName:ddb4cb56-7904-4742-ac26-ecbd4f38df78 nodeName:}" failed. No retries permitted until 2026-05-20 15:31:56.672737527 +0000 UTC m=+6.376831098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-682mt" (UniqueName: "kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt") pod "network-check-target-q4gpd" (UID: "ddb4cb56-7904-4742-ac26-ecbd4f38df78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:55.052802 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:55.052726    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:55.052947 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:55.052868    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:31:55.053307 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:55.053289    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:31:55.053401 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:55.053382    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:31:55.230853 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:55.230818    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal" event={"ID":"a068ba9a88bc229cfafccce16e49d8ae","Type":"ContainerStarted","Data":"81306279121fe787c9b4367bd5f27a6760c95d96d90baab324e35b8eb369874c"}
May 20 15:31:55.247256 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:55.247200    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-227.ec2.internal" podStartSLOduration=3.247180638 podStartE2EDuration="3.247180638s" podCreationTimestamp="2026-05-20 15:31:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:31:54.212323565 +0000 UTC m=+3.916417156" watchObservedRunningTime="2026-05-20 15:31:55.247180638 +0000 UTC m=+4.951274229"
May 20 15:31:55.247842 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:55.247794    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-227.ec2.internal" podStartSLOduration=3.247783815 podStartE2EDuration="3.247783815s" podCreationTimestamp="2026-05-20 15:31:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:31:55.246026515 +0000 UTC m=+4.950120105" watchObservedRunningTime="2026-05-20 15:31:55.247783815 +0000 UTC m=+4.951877408"
May 20 15:31:56.587456 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:56.587420    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:56.587926 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:56.587573    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:56.587926 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:56.587679    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs podName:9f8d9d11-6bdb-402e-ab74-7485d6c52a79 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:00.587660495 +0000 UTC m=+10.291754065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs") pod "network-metrics-daemon-46dsm" (UID: "9f8d9d11-6bdb-402e-ab74-7485d6c52a79") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:31:56.688269 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:56.688232    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-682mt\" (UniqueName: \"kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt\") pod \"network-check-target-q4gpd\" (UID: \"ddb4cb56-7904-4742-ac26-ecbd4f38df78\") " pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:31:56.688449 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:56.688417    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:31:56.688449 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:56.688432    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:31:56.688449 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:56.688442    2567 projected.go:194] Error preparing data for projected volume kube-api-access-682mt for pod openshift-network-diagnostics/network-check-target-q4gpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:56.688653 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:56.688482    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt podName:ddb4cb56-7904-4742-ac26-ecbd4f38df78 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:00.688468603 +0000 UTC m=+10.392562172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-682mt" (UniqueName: "kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt") pod "network-check-target-q4gpd" (UID: "ddb4cb56-7904-4742-ac26-ecbd4f38df78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:31:57.053208 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:57.053120    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:31:57.053208 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:57.053174    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:57.053446 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:57.053264    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:31:57.053446 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:57.053400    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:31:59.053784 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:59.053111    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:31:59.053784 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:59.053239    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:31:59.053784 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:31:59.053640    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:31:59.053784 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:31:59.053739    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:00.621340 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:00.621303    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:00.621799 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:00.621441    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:00.621799 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:00.621509    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs podName:9f8d9d11-6bdb-402e-ab74-7485d6c52a79 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:08.621487586 +0000 UTC m=+18.325581243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs") pod "network-metrics-daemon-46dsm" (UID: "9f8d9d11-6bdb-402e-ab74-7485d6c52a79") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:00.722708 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:00.722138    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-682mt\" (UniqueName: \"kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt\") pod \"network-check-target-q4gpd\" (UID: \"ddb4cb56-7904-4742-ac26-ecbd4f38df78\") " pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:00.722708 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:00.722293    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:00.722708 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:00.722316    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:00.722708 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:00.722330    2567 projected.go:194] Error preparing data for projected volume kube-api-access-682mt for pod openshift-network-diagnostics/network-check-target-q4gpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:00.722708 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:00.722390    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt podName:ddb4cb56-7904-4742-ac26-ecbd4f38df78 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:08.722371441 +0000 UTC m=+18.426465028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-682mt" (UniqueName: "kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt") pod "network-check-target-q4gpd" (UID: "ddb4cb56-7904-4742-ac26-ecbd4f38df78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:01.053255 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:01.053220    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:01.053433 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:01.053343    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:01.053599 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:01.053554    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:01.053713 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:01.053690    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:03.052814 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:03.052772    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:03.053297 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:03.052784    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:03.053297 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:03.052918    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:03.053297 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:03.053022    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:05.052277 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.052247    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:05.052737 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.052248    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:05.052737 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:05.052358    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:05.052737 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:05.052431    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:05.373847 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.373810    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-dpk92"]
May 20 15:32:05.376012 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.375989    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dpk92"
May 20 15:32:05.381099 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.381073    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 20 15:32:05.381371 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.381350    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 20 15:32:05.381577 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.381553    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w9tn4\""
May 20 15:32:05.457323 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.457279    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ab595d7-4e45-464c-87e7-58d498cbb575-hosts-file\") pod \"node-resolver-dpk92\" (UID: \"0ab595d7-4e45-464c-87e7-58d498cbb575\") " pod="openshift-dns/node-resolver-dpk92"
May 20 15:32:05.457323 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.457315    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66zz\" (UniqueName: \"kubernetes.io/projected/0ab595d7-4e45-464c-87e7-58d498cbb575-kube-api-access-h66zz\") pod \"node-resolver-dpk92\" (UID: \"0ab595d7-4e45-464c-87e7-58d498cbb575\") " pod="openshift-dns/node-resolver-dpk92"
May 20 15:32:05.457522 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.457351    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab595d7-4e45-464c-87e7-58d498cbb575-tmp-dir\") pod \"node-resolver-dpk92\" (UID: \"0ab595d7-4e45-464c-87e7-58d498cbb575\") " pod="openshift-dns/node-resolver-dpk92"
May 20 15:32:05.557777 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.557741    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ab595d7-4e45-464c-87e7-58d498cbb575-hosts-file\") pod \"node-resolver-dpk92\" (UID: \"0ab595d7-4e45-464c-87e7-58d498cbb575\") " pod="openshift-dns/node-resolver-dpk92"
May 20 15:32:05.557941 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.557784    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h66zz\" (UniqueName: \"kubernetes.io/projected/0ab595d7-4e45-464c-87e7-58d498cbb575-kube-api-access-h66zz\") pod \"node-resolver-dpk92\" (UID: \"0ab595d7-4e45-464c-87e7-58d498cbb575\") " pod="openshift-dns/node-resolver-dpk92"
May 20 15:32:05.557941 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.557830    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab595d7-4e45-464c-87e7-58d498cbb575-tmp-dir\") pod \"node-resolver-dpk92\" (UID: \"0ab595d7-4e45-464c-87e7-58d498cbb575\") " pod="openshift-dns/node-resolver-dpk92"
May 20 15:32:05.557941 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.557876    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ab595d7-4e45-464c-87e7-58d498cbb575-hosts-file\") pod \"node-resolver-dpk92\" (UID: \"0ab595d7-4e45-464c-87e7-58d498cbb575\") " pod="openshift-dns/node-resolver-dpk92"
May 20 15:32:05.558236 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.558194    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab595d7-4e45-464c-87e7-58d498cbb575-tmp-dir\") pod \"node-resolver-dpk92\" (UID: \"0ab595d7-4e45-464c-87e7-58d498cbb575\") " pod="openshift-dns/node-resolver-dpk92"
May 20 15:32:05.569223 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.569192    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66zz\" (UniqueName: \"kubernetes.io/projected/0ab595d7-4e45-464c-87e7-58d498cbb575-kube-api-access-h66zz\") pod \"node-resolver-dpk92\" (UID: \"0ab595d7-4e45-464c-87e7-58d498cbb575\") " pod="openshift-dns/node-resolver-dpk92"
May 20 15:32:05.687066 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:05.686988    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dpk92"
May 20 15:32:07.052550 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:07.052467    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:07.052550 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:07.052512    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:07.052997 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:07.052631    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:07.052997 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:07.052720    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:08.679231 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:08.679195    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:08.679618 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:08.679364    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:08.679618 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:08.679433    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs podName:9f8d9d11-6bdb-402e-ab74-7485d6c52a79 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:24.6794179 +0000 UTC m=+34.383511471 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs") pod "network-metrics-daemon-46dsm" (UID: "9f8d9d11-6bdb-402e-ab74-7485d6c52a79") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:08.779845 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:08.779748    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-682mt\" (UniqueName: \"kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt\") pod \"network-check-target-q4gpd\" (UID: \"ddb4cb56-7904-4742-ac26-ecbd4f38df78\") " pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:08.780002 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:08.779912    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:08.780002 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:08.779931    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:08.780002 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:08.779940    2567 projected.go:194] Error preparing data for projected volume kube-api-access-682mt for pod openshift-network-diagnostics/network-check-target-q4gpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:08.780002 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:08.779994    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt podName:ddb4cb56-7904-4742-ac26-ecbd4f38df78 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:24.779980747 +0000 UTC m=+34.484074315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-682mt" (UniqueName: "kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt") pod "network-check-target-q4gpd" (UID: "ddb4cb56-7904-4742-ac26-ecbd4f38df78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:09.052819 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:09.052728    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:09.052963 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:09.052866    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:09.052963 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:09.052913    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:09.053064 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:09.053002    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:11.052852 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:11.052816    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:11.053233 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:11.052908    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:11.053233 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:11.052994    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:11.053233 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:11.053082    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:11.370211 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:11.370180    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab595d7_4e45_464c_87e7_58d498cbb575.slice/crio-fb67f6158edb0604580cd1d2048b431e60cfe6ee45b2abd4975d395ee9f4246d WatchSource:0}: Error finding container fb67f6158edb0604580cd1d2048b431e60cfe6ee45b2abd4975d395ee9f4246d: Status 404 returned error can't find the container with id fb67f6158edb0604580cd1d2048b431e60cfe6ee45b2abd4975d395ee9f4246d
May 20 15:32:12.262631 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.262277    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:32:12.263498 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.262938    2567 generic.go:358] "Generic (PLEG): container finished" podID="581482dd-fca5-4834-99f4-519a6df326fb" containerID="62a63a6c07e16bef81ce41b803a509908f93633419236c3a41e36be7e057d886" exitCode=1
May 20 15:32:12.263498 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.262996    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx" event={"ID":"581482dd-fca5-4834-99f4-519a6df326fb","Type":"ContainerStarted","Data":"f4a004866a2d65e93185040021789437eb083d3b9bb3996f91702f876ad8588c"}
May 20 15:32:12.263498 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.263039    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx" event={"ID":"581482dd-fca5-4834-99f4-519a6df326fb","Type":"ContainerStarted","Data":"e8cb0916b480e6a0aef840f334f075da5c662043da457ed76395ec36afeedab7"}
May 20 15:32:12.263498 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.263054    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx" event={"ID":"581482dd-fca5-4834-99f4-519a6df326fb","Type":"ContainerStarted","Data":"97b81d45a17acb4c7a8bc92b2fb455bd499a0db343aff3c7887ab7ba084491ca"}
May 20 15:32:12.263498 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.263066    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx" event={"ID":"581482dd-fca5-4834-99f4-519a6df326fb","Type":"ContainerStarted","Data":"2f3a595a7cb99a3108eedba649d51f2d33008e23a871238e5c0864c291ce62e5"}
May 20 15:32:12.263498 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.263078    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx" event={"ID":"581482dd-fca5-4834-99f4-519a6df326fb","Type":"ContainerDied","Data":"62a63a6c07e16bef81ce41b803a509908f93633419236c3a41e36be7e057d886"}
May 20 15:32:12.263498 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.263093    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx" event={"ID":"581482dd-fca5-4834-99f4-519a6df326fb","Type":"ContainerStarted","Data":"4dd9736e0b3195e7fcc5f163d8e692f5ac9b762935c189d8b83fd51ab78798f1"}
May 20 15:32:12.264518 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.264493    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5" event={"ID":"04986e91-b72e-47f8-a17f-c6390009d60b","Type":"ContainerStarted","Data":"ee48e1750d22ddf26bb87ded452e14a9b1fe8c6a848dd34fa217c0871f8c510e"}
May 20 15:32:12.265738 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.265708    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b25vj" event={"ID":"71f15969-7e79-4580-b24d-41f479c3715e","Type":"ContainerStarted","Data":"cd62ba33896a68958acba308a4383397cfe572705992b7822c834fe1fcd5cabb"}
May 20 15:32:12.267206 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.267180    2567 generic.go:358] "Generic (PLEG): container finished" podID="e367b95c-7b4d-4d9c-8a71-1429e26b0230" containerID="94758561e61ca209bac3ffd7604932c28a3007bb277917f1a76a879590e3e3bb" exitCode=0
May 20 15:32:12.267306 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.267260    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsxb5" event={"ID":"e367b95c-7b4d-4d9c-8a71-1429e26b0230","Type":"ContainerDied","Data":"94758561e61ca209bac3ffd7604932c28a3007bb277917f1a76a879590e3e3bb"}
May 20 15:32:12.268719 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.268658    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dpk92" event={"ID":"0ab595d7-4e45-464c-87e7-58d498cbb575","Type":"ContainerStarted","Data":"38cb406e894a3f08f5e72fc124af1ed47c896aeda8edcc461ef293b7bc1d5bc7"}
May 20 15:32:12.268719 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.268689    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dpk92" event={"ID":"0ab595d7-4e45-464c-87e7-58d498cbb575","Type":"ContainerStarted","Data":"fb67f6158edb0604580cd1d2048b431e60cfe6ee45b2abd4975d395ee9f4246d"}
May 20 15:32:12.270785 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.270760    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ccd2v" event={"ID":"ffa2ed51-616b-4441-a33e-45c761dd075e","Type":"ContainerStarted","Data":"2fb14a54f22546b3921afcbc6ecbb3bdb95ecf71476e37457bcb4125cb3b4c50"}
May 20 15:32:12.272073 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.272051    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mpmsj" event={"ID":"f65cd891-4c27-4070-972c-9a97f6fc62af","Type":"ContainerStarted","Data":"b8e079b147c793b4a18794697c11c4d19f127d8ee31e19feb3c20b34549b2916"}
May 20 15:32:12.274158 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.274131    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4t7b8" event={"ID":"6b36f593-5d2c-45dc-b0ff-bd5e90381257","Type":"ContainerStarted","Data":"36afa19482698f3a44b47ed650f2f41f3ef3c8820f0d037c370fb69a49c6e1c8"}
May 20 15:32:12.289723 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.289672    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b25vj" podStartSLOduration=8.22767558 podStartE2EDuration="21.289658053s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:53.533646151 +0000 UTC m=+3.237739720" lastFinishedPulling="2026-05-20 15:32:06.595628623 +0000 UTC m=+16.299722193" observedRunningTime="2026-05-20 15:32:12.289096924 +0000 UTC m=+21.993190516" watchObservedRunningTime="2026-05-20 15:32:12.289658053 +0000 UTC m=+21.993751642"
May 20 15:32:12.330035 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.329973    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-dpk92" podStartSLOduration=7.329957928 podStartE2EDuration="7.329957928s" podCreationTimestamp="2026-05-20 15:32:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:32:12.329469687 +0000 UTC m=+22.033563278" watchObservedRunningTime="2026-05-20 15:32:12.329957928 +0000 UTC m=+22.034051518"
May 20 15:32:12.346968 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.346912    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ccd2v" podStartSLOduration=3.5251502390000002 podStartE2EDuration="21.346893975s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:53.540560004 +0000 UTC m=+3.244653571" lastFinishedPulling="2026-05-20 15:32:11.362303724 +0000 UTC m=+21.066397307" observedRunningTime="2026-05-20 15:32:12.346878932 +0000 UTC m=+22.050972523" watchObservedRunningTime="2026-05-20 15:32:12.346893975 +0000 UTC m=+22.050987566"
May 20 15:32:12.367063 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.367013    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mpmsj" podStartSLOduration=3.5432175580000003 podStartE2EDuration="21.366994741s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:53.540377791 +0000 UTC m=+3.244471374" lastFinishedPulling="2026-05-20 15:32:11.364154974 +0000 UTC m=+21.068248557" observedRunningTime="2026-05-20 15:32:12.36634769 +0000 UTC m=+22.070441271" watchObservedRunningTime="2026-05-20 15:32:12.366994741 +0000 UTC m=+22.071088333"
May 20 15:32:12.775887 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:12.775858    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 20 15:32:13.030388 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:13.030212    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-20T15:32:12.775880263Z","UUID":"ea81e196-49b5-4409-950f-4c5e0e5148bd","Handler":null,"Name":"","Endpoint":""}
May 20 15:32:13.032019 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:13.031987    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 20 15:32:13.032174 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:13.032026    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 20 15:32:13.052856 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:13.052808    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:13.055928 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:13.053739    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:13.055928 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:13.053861    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:13.055928 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:13.053963    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:13.277850 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:13.277807    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5" event={"ID":"04986e91-b72e-47f8-a17f-c6390009d60b","Type":"ContainerStarted","Data":"2db195cc95038029d975bb69da8c6f9cf12e338afe72339456cf79c7bf5c0743"}
May 20 15:32:13.280704 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:13.280634    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sb4fl" event={"ID":"d6731423-938b-453d-965d-db728cfa0efb","Type":"ContainerStarted","Data":"65dce2e90872bc7400896b6d8766ba0b3742f923ad129b897fb822708756c4b0"}
May 20 15:32:13.299051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:13.298287    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-4t7b8" podStartSLOduration=4.456807126 podStartE2EDuration="22.298268385s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:53.537942423 +0000 UTC m=+3.242036006" lastFinishedPulling="2026-05-20 15:32:11.379403697 +0000 UTC m=+21.083497265" observedRunningTime="2026-05-20 15:32:12.387525839 +0000 UTC m=+22.091619431" watchObservedRunningTime="2026-05-20 15:32:13.298268385 +0000 UTC m=+23.002361976"
May 20 15:32:13.299051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:13.298903    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sb4fl" podStartSLOduration=4.478439529 podStartE2EDuration="22.298892321s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:53.541843136 +0000 UTC m=+3.245936706" lastFinishedPulling="2026-05-20 15:32:11.362295914 +0000 UTC m=+21.066389498" observedRunningTime="2026-05-20 15:32:13.298054979 +0000 UTC m=+23.002148569" watchObservedRunningTime="2026-05-20 15:32:13.298892321 +0000 UTC m=+23.002985912"
May 20 15:32:13.936107 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:13.936064    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b25vj"
May 20 15:32:13.936794 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:13.936770    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b25vj"
May 20 15:32:14.284734 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:14.283996    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5" event={"ID":"04986e91-b72e-47f8-a17f-c6390009d60b","Type":"ContainerStarted","Data":"d53ad17714bbe325cd70192135696d70b4554e10662192d7842cfd524cd21bcc"}
May 20 15:32:14.284734 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:14.284336    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b25vj"
May 20 15:32:14.284734 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:14.284697    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b25vj"
May 20 15:32:14.306536 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:14.306475    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwss5" podStartSLOduration=2.935538696 podStartE2EDuration="23.306459896s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:53.535291334 +0000 UTC m=+3.239384902" lastFinishedPulling="2026-05-20 15:32:13.906212523 +0000 UTC m=+23.610306102" observedRunningTime="2026-05-20 15:32:14.305268031 +0000 UTC m=+24.009361620" watchObservedRunningTime="2026-05-20 15:32:14.306459896 +0000 UTC m=+24.010553468"
May 20 15:32:15.052267 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:15.052222    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:15.052476 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:15.052339    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:15.052476 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:15.052430    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:15.052618 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:15.052540    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:15.288854 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:15.288825    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:32:15.289324 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:15.289211    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx" event={"ID":"581482dd-fca5-4834-99f4-519a6df326fb","Type":"ContainerStarted","Data":"8f95c91bc29fb2d92917447bd8f1faa96071707b6114be135e33bf48ab921c34"}
May 20 15:32:17.052877 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.052602    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:17.052877 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.052602    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:17.052877 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:17.052872    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:17.053644 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:17.052955    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:17.295725 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.295696    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:32:17.296095 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.296064    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx" event={"ID":"581482dd-fca5-4834-99f4-519a6df326fb","Type":"ContainerStarted","Data":"b330f666b1d10d76b195f978f5185afc561365f976f188831195cfc89ec8f25c"}
May 20 15:32:17.296568 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.296544    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:32:17.296568 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.296571    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:32:17.296722 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.296547    2567 scope.go:117] "RemoveContainer" containerID="62a63a6c07e16bef81ce41b803a509908f93633419236c3a41e36be7e057d886"
May 20 15:32:17.298220 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.298195    2567 generic.go:358] "Generic (PLEG): container finished" podID="e367b95c-7b4d-4d9c-8a71-1429e26b0230" containerID="b430a5d880ca9b3492ee2d3c8316257002b5bde29ecd94c2f73beb4adc77db2c" exitCode=0
May 20 15:32:17.298338 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.298246    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsxb5" event={"ID":"e367b95c-7b4d-4d9c-8a71-1429e26b0230","Type":"ContainerDied","Data":"b430a5d880ca9b3492ee2d3c8316257002b5bde29ecd94c2f73beb4adc77db2c"}
May 20 15:32:17.314419 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.314394    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:32:17.314628 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.314614    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:32:17.421114 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:17.421080    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dpk92_0ab595d7-4e45-464c-87e7-58d498cbb575/dns-node-resolver/0.log"
May 20 15:32:18.207147 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:18.207122    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ccd2v_ffa2ed51-616b-4441-a33e-45c761dd075e/node-ca/0.log"
May 20 15:32:18.301889 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:18.301798    2567 generic.go:358] "Generic (PLEG): container finished" podID="e367b95c-7b4d-4d9c-8a71-1429e26b0230" containerID="aeb6355a98bd8248699bedfe085abcdaaf061ffb7fbb2fe58fb01cf4e5ade7f6" exitCode=0
May 20 15:32:18.302033 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:18.301889    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsxb5" event={"ID":"e367b95c-7b4d-4d9c-8a71-1429e26b0230","Type":"ContainerDied","Data":"aeb6355a98bd8248699bedfe085abcdaaf061ffb7fbb2fe58fb01cf4e5ade7f6"}
May 20 15:32:18.305240 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:18.305221    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:32:18.305575 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:18.305554    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx" event={"ID":"581482dd-fca5-4834-99f4-519a6df326fb","Type":"ContainerStarted","Data":"90e3371be9000ac8e48d03beccf8f38fb34ecef4776c09b73cd7fd449a132094"}
May 20 15:32:18.305725 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:18.305710    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 20 15:32:18.382281 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:18.382229    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx" podStartSLOduration=9.498871142 podStartE2EDuration="27.382210632s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:53.528989011 +0000 UTC m=+3.233082582" lastFinishedPulling="2026-05-20 15:32:11.412328487 +0000 UTC m=+21.116422072" observedRunningTime="2026-05-20 15:32:18.382058572 +0000 UTC m=+28.086152164" watchObservedRunningTime="2026-05-20 15:32:18.382210632 +0000 UTC m=+28.086304221"
May 20 15:32:18.730007 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:18.729970    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q4gpd"]
May 20 15:32:18.730205 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:18.730126    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:18.730303 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:18.730204    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:18.732709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:18.732681    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-46dsm"]
May 20 15:32:18.732835 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:18.732822    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:18.732985 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:18.732963    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:19.309418 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:19.309204    2567 generic.go:358] "Generic (PLEG): container finished" podID="e367b95c-7b4d-4d9c-8a71-1429e26b0230" containerID="c7a68a78a60387dc8b0c7e92c5b1df8af6e594ec8917cbca3b027eedce3d5b48" exitCode=0
May 20 15:32:19.309801 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:19.309267    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsxb5" event={"ID":"e367b95c-7b4d-4d9c-8a71-1429e26b0230","Type":"ContainerDied","Data":"c7a68a78a60387dc8b0c7e92c5b1df8af6e594ec8917cbca3b027eedce3d5b48"}
May 20 15:32:19.309801 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:19.309610    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 20 15:32:20.052797 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:20.052760    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:20.052998 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:20.052895    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:21.053361 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:21.053272    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:21.053875 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:21.053381    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:22.053042 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:22.053004    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:22.053223 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:22.053141    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:22.570035 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:22.570000    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:32:22.570572 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:22.570263    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
May 20 15:32:22.583242 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:22.583215    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dbwqx"
May 20 15:32:23.052602 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:23.052544    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:23.052764 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:23.052711    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:24.052377 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:24.052334    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:24.052818 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:24.052474    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:24.690096 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:24.690061    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:24.690275 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:24.690189    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:24.690322 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:24.690276    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs podName:9f8d9d11-6bdb-402e-ab74-7485d6c52a79 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:56.690252032 +0000 UTC m=+66.394345614 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs") pod "network-metrics-daemon-46dsm" (UID: "9f8d9d11-6bdb-402e-ab74-7485d6c52a79") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 20 15:32:24.790832 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:24.790795    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-682mt\" (UniqueName: \"kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt\") pod \"network-check-target-q4gpd\" (UID: \"ddb4cb56-7904-4742-ac26-ecbd4f38df78\") " pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:24.790996 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:24.790943    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 20 15:32:24.790996 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:24.790958    2567 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 20 15:32:24.790996 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:24.790967    2567 projected.go:194] Error preparing data for projected volume kube-api-access-682mt for pod openshift-network-diagnostics/network-check-target-q4gpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:24.791114 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:24.791019    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt podName:ddb4cb56-7904-4742-ac26-ecbd4f38df78 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:56.791003487 +0000 UTC m=+66.495097075 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-682mt" (UniqueName: "kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt") pod "network-check-target-q4gpd" (UID: "ddb4cb56-7904-4742-ac26-ecbd4f38df78") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 20 15:32:25.052057 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:25.052031    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:25.052190 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:25.052133    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:25.325057 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:25.325024    2567 generic.go:358] "Generic (PLEG): container finished" podID="e367b95c-7b4d-4d9c-8a71-1429e26b0230" containerID="2909c17f7ec770a1499207002d37b8d488eef2b4e0c9891a15a966beaec5152e" exitCode=0
May 20 15:32:25.325511 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:25.325074    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsxb5" event={"ID":"e367b95c-7b4d-4d9c-8a71-1429e26b0230","Type":"ContainerDied","Data":"2909c17f7ec770a1499207002d37b8d488eef2b4e0c9891a15a966beaec5152e"}
May 20 15:32:26.052804 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:26.052761    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:26.053048 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:26.052881    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:26.329466 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:26.329387    2567 generic.go:358] "Generic (PLEG): container finished" podID="e367b95c-7b4d-4d9c-8a71-1429e26b0230" containerID="31ecf9eba8be1cc12531a734d943d6aa87175c2332b09820b9ce9d2eb19874a3" exitCode=0
May 20 15:32:26.329466 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:26.329427    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsxb5" event={"ID":"e367b95c-7b4d-4d9c-8a71-1429e26b0230","Type":"ContainerDied","Data":"31ecf9eba8be1cc12531a734d943d6aa87175c2332b09820b9ce9d2eb19874a3"}
May 20 15:32:27.053098 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:27.053056    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:27.053276 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:27.053186    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:27.333919 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:27.333832    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fsxb5" event={"ID":"e367b95c-7b4d-4d9c-8a71-1429e26b0230","Type":"ContainerStarted","Data":"dba717696f286f7486d5f29f0d5e3003f39c7a25f3b425aa33a14c10ab278def"}
May 20 15:32:27.357080 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:27.357034    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fsxb5" podStartSLOduration=4.838846971 podStartE2EDuration="36.3570191s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:31:53.531471206 +0000 UTC m=+3.235564774" lastFinishedPulling="2026-05-20 15:32:25.049643335 +0000 UTC m=+34.753736903" observedRunningTime="2026-05-20 15:32:27.356734562 +0000 UTC m=+37.060828153" watchObservedRunningTime="2026-05-20 15:32:27.3570191 +0000 UTC m=+37.061112690"
May 20 15:32:28.052796 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:28.052745    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:28.052964 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:28.052850    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:29.053035 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:29.052751    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:29.053499 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:29.053131    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:30.052530 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:30.052495    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:30.052735 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:30.052623    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:31.053778 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:31.053738    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:31.054173 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:31.053874    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:32.052750 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:32.052708    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:32.052931 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:32.052818    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:33.052670 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:33.052633    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:33.053149 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:33.052743    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:34.052258 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:34.052223    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:34.052441 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:34.052327    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:35.052459 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:35.052427    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:35.052850 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:35.052536    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:36.052752 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:36.052715    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:36.053139 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:36.052820    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:37.052177 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:37.052144    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:37.052351 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:37.052283    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:38.052751 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:38.052717    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:38.053137 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:38.052823    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:39.052288 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:39.052249    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:39.052455 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:39.052399    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:40.052861 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:40.052819    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:40.053241 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:40.052925    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:41.053217 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:41.053182    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:41.053609 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:41.053283    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-46dsm" podUID="9f8d9d11-6bdb-402e-ab74-7485d6c52a79"
May 20 15:32:42.052697 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.052659    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:42.052946 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:42.052765    2567 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4gpd" podUID="ddb4cb56-7904-4742-ac26-ecbd4f38df78"
May 20 15:32:42.636065 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.636037    2567 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-130-227.ec2.internal" event="NodeReady"
May 20 15:32:42.636449 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.636161    2567 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 20 15:32:42.739414 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.739380    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2gp62"]
May 20 15:32:42.786743 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.786712    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x555j"]
May 20 15:32:42.786936 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.786889    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2gp62"
May 20 15:32:42.793310 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.793287    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5s49c\""
May 20 15:32:42.793544 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.793527    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 20 15:32:42.793809 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.793790    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 20 15:32:42.793885 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.793801    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 20 15:32:42.807220 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.807193    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2gp62"]
May 20 15:32:42.807220 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.807221    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x555j"]
May 20 15:32:42.807220 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.807236    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p574b"]
May 20 15:32:42.807440 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.807363    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x555j"
May 20 15:32:42.810551 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.810532    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zngvt\""
May 20 15:32:42.810783 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.810764    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 20 15:32:42.811017 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.811003    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 20 15:32:42.827697 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.827676    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p574b"]
May 20 15:32:42.827837 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.827786    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:42.834370 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.834346    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 20 15:32:42.834499 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.834378    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 20 15:32:42.834499 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.834418    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 20 15:32:42.834499 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.834462    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b28xv\""
May 20 15:32:42.834499 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.834466    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 20 15:32:42.916322 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.916235    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67023ceb-0ace-44c4-9111-eec6cfaf0212-tmp-dir\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:42.916322 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.916290    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvb9r\" (UniqueName: \"kubernetes.io/projected/67023ceb-0ace-44c4-9111-eec6cfaf0212-kube-api-access-nvb9r\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:42.916493 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.916352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67023ceb-0ace-44c4-9111-eec6cfaf0212-metrics-tls\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:42.916493 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.916385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d481c73-56b1-4d54-97e3-9344490303ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:42.916493 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.916425    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d481c73-56b1-4d54-97e3-9344490303ee-data-volume\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:42.916493 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.916442    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs8rv\" (UniqueName: \"kubernetes.io/projected/8d481c73-56b1-4d54-97e3-9344490303ee-kube-api-access-gs8rv\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:42.916646 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.916494    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r27px\" (UniqueName: \"kubernetes.io/projected/2a9b92fe-cce1-4050-85b1-1f4b12e178b6-kube-api-access-r27px\") pod \"ingress-canary-2gp62\" (UID: \"2a9b92fe-cce1-4050-85b1-1f4b12e178b6\") " pod="openshift-ingress-canary/ingress-canary-2gp62"
May 20 15:32:42.916646 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.916530    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d481c73-56b1-4d54-97e3-9344490303ee-crio-socket\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:42.916646 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.916549    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67023ceb-0ace-44c4-9111-eec6cfaf0212-config-volume\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:42.916646 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.916603    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d481c73-56b1-4d54-97e3-9344490303ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:42.916646 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:42.916641    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a9b92fe-cce1-4050-85b1-1f4b12e178b6-cert\") pod \"ingress-canary-2gp62\" (UID: \"2a9b92fe-cce1-4050-85b1-1f4b12e178b6\") " pod="openshift-ingress-canary/ingress-canary-2gp62"
May 20 15:32:43.017758 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.017719    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67023ceb-0ace-44c4-9111-eec6cfaf0212-metrics-tls\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:43.017758 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.017758    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d481c73-56b1-4d54-97e3-9344490303ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:43.017922 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.017785    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d481c73-56b1-4d54-97e3-9344490303ee-data-volume\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:43.018019 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.017994    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8rv\" (UniqueName: \"kubernetes.io/projected/8d481c73-56b1-4d54-97e3-9344490303ee-kube-api-access-gs8rv\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:43.018067 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018042    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r27px\" (UniqueName: \"kubernetes.io/projected/2a9b92fe-cce1-4050-85b1-1f4b12e178b6-kube-api-access-r27px\") pod \"ingress-canary-2gp62\" (UID: \"2a9b92fe-cce1-4050-85b1-1f4b12e178b6\") " pod="openshift-ingress-canary/ingress-canary-2gp62"
May 20 15:32:43.018118 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018071    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d481c73-56b1-4d54-97e3-9344490303ee-crio-socket\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:43.018118 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018074    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d481c73-56b1-4d54-97e3-9344490303ee-data-volume\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:43.018118 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018096    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67023ceb-0ace-44c4-9111-eec6cfaf0212-config-volume\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:43.018253 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018218    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d481c73-56b1-4d54-97e3-9344490303ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:43.018305 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018270    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a9b92fe-cce1-4050-85b1-1f4b12e178b6-cert\") pod \"ingress-canary-2gp62\" (UID: \"2a9b92fe-cce1-4050-85b1-1f4b12e178b6\") " pod="openshift-ingress-canary/ingress-canary-2gp62"
May 20 15:32:43.018305 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018259    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d481c73-56b1-4d54-97e3-9344490303ee-crio-socket\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:43.018305 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018298    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67023ceb-0ace-44c4-9111-eec6cfaf0212-tmp-dir\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:43.018641 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018618    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67023ceb-0ace-44c4-9111-eec6cfaf0212-config-volume\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:43.018754 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018655    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67023ceb-0ace-44c4-9111-eec6cfaf0212-tmp-dir\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:43.018754 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018705    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvb9r\" (UniqueName: \"kubernetes.io/projected/67023ceb-0ace-44c4-9111-eec6cfaf0212-kube-api-access-nvb9r\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:43.018754 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.018722    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d481c73-56b1-4d54-97e3-9344490303ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:43.022047 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.022026    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d481c73-56b1-4d54-97e3-9344490303ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:43.022047 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.022037    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67023ceb-0ace-44c4-9111-eec6cfaf0212-metrics-tls\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:43.022168 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.022040    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a9b92fe-cce1-4050-85b1-1f4b12e178b6-cert\") pod \"ingress-canary-2gp62\" (UID: \"2a9b92fe-cce1-4050-85b1-1f4b12e178b6\") " pod="openshift-ingress-canary/ingress-canary-2gp62"
May 20 15:32:43.050936 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.050908    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r27px\" (UniqueName: \"kubernetes.io/projected/2a9b92fe-cce1-4050-85b1-1f4b12e178b6-kube-api-access-r27px\") pod \"ingress-canary-2gp62\" (UID: \"2a9b92fe-cce1-4050-85b1-1f4b12e178b6\") " pod="openshift-ingress-canary/ingress-canary-2gp62"
May 20 15:32:43.052911 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.052892    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:43.056905 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.056886    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hm9j9\""
May 20 15:32:43.057126 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.057114    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 15:32:43.081675 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.081642    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs8rv\" (UniqueName: \"kubernetes.io/projected/8d481c73-56b1-4d54-97e3-9344490303ee-kube-api-access-gs8rv\") pod \"insights-runtime-extractor-p574b\" (UID: \"8d481c73-56b1-4d54-97e3-9344490303ee\") " pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:43.084715 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.084695    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvb9r\" (UniqueName: \"kubernetes.io/projected/67023ceb-0ace-44c4-9111-eec6cfaf0212-kube-api-access-nvb9r\") pod \"dns-default-x555j\" (UID: \"67023ceb-0ace-44c4-9111-eec6cfaf0212\") " pod="openshift-dns/dns-default-x555j"
May 20 15:32:43.096571 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.096550    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2gp62"
May 20 15:32:43.116274 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.116243    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x555j"
May 20 15:32:43.136469 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.136127    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p574b"
May 20 15:32:43.282654 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.282620    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2gp62"]
May 20 15:32:43.286462 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:43.286428    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9b92fe_cce1_4050_85b1_1f4b12e178b6.slice/crio-f463a627b2512b0752e4ebfcd63ed1539ad61e22468ef5c5f010b2b7d5b084e5 WatchSource:0}: Error finding container f463a627b2512b0752e4ebfcd63ed1539ad61e22468ef5c5f010b2b7d5b084e5: Status 404 returned error can't find the container with id f463a627b2512b0752e4ebfcd63ed1539ad61e22468ef5c5f010b2b7d5b084e5
May 20 15:32:43.307328 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.307301    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x555j"]
May 20 15:32:43.310625 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:43.310569    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67023ceb_0ace_44c4_9111_eec6cfaf0212.slice/crio-34057117daab3485c4a871ffe3af7e66e282a9fcf008ed74f08faec2ae666e99 WatchSource:0}: Error finding container 34057117daab3485c4a871ffe3af7e66e282a9fcf008ed74f08faec2ae666e99: Status 404 returned error can't find the container with id 34057117daab3485c4a871ffe3af7e66e282a9fcf008ed74f08faec2ae666e99
May 20 15:32:43.311325 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.311285    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p574b"]
May 20 15:32:43.313997 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:43.313965    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d481c73_56b1_4d54_97e3_9344490303ee.slice/crio-530cc0f927d146a5d49ec078dfb7d106c4faed3394ed927a400ca41724e2f506 WatchSource:0}: Error finding container 530cc0f927d146a5d49ec078dfb7d106c4faed3394ed927a400ca41724e2f506: Status 404 returned error can't find the container with id 530cc0f927d146a5d49ec078dfb7d106c4faed3394ed927a400ca41724e2f506
May 20 15:32:43.363123 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.363081    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p574b" event={"ID":"8d481c73-56b1-4d54-97e3-9344490303ee","Type":"ContainerStarted","Data":"530cc0f927d146a5d49ec078dfb7d106c4faed3394ed927a400ca41724e2f506"}
May 20 15:32:43.364146 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.364116    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x555j" event={"ID":"67023ceb-0ace-44c4-9111-eec6cfaf0212","Type":"ContainerStarted","Data":"34057117daab3485c4a871ffe3af7e66e282a9fcf008ed74f08faec2ae666e99"}
May 20 15:32:43.366457 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:43.366430    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2gp62" event={"ID":"2a9b92fe-cce1-4050-85b1-1f4b12e178b6","Type":"ContainerStarted","Data":"f463a627b2512b0752e4ebfcd63ed1539ad61e22468ef5c5f010b2b7d5b084e5"}
May 20 15:32:44.053127 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:44.053085    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:44.056909 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:44.056878    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 15:32:44.057057 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:44.056919    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9zz48\""
May 20 15:32:44.057899 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:44.057877    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 15:32:44.369637 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:44.369602    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p574b" event={"ID":"8d481c73-56b1-4d54-97e3-9344490303ee","Type":"ContainerStarted","Data":"949a0722e8b89762d3b59ca2ebebd78e080b951007a4d4bab22b70f6aa408195"}
May 20 15:32:46.378037 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.377998    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p574b" event={"ID":"8d481c73-56b1-4d54-97e3-9344490303ee","Type":"ContainerStarted","Data":"bf2a70e2308939f74d134b9ea61f76362c308a9216049316173528ca507f87a8"}
May 20 15:32:46.379770 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.379738    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x555j" event={"ID":"67023ceb-0ace-44c4-9111-eec6cfaf0212","Type":"ContainerStarted","Data":"78a07032f28b1c297ab1ac730d955de43d577f5bb60930803c2980e8d5f1e5cf"}
May 20 15:32:46.380064 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.379778    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x555j" event={"ID":"67023ceb-0ace-44c4-9111-eec6cfaf0212","Type":"ContainerStarted","Data":"1d24de2306205ea67b0ade71d71ead461fac82616469bec0174956b7a4f336dd"}
May 20 15:32:46.380064 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.379840    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x555j"
May 20 15:32:46.381159 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.381132    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2gp62" event={"ID":"2a9b92fe-cce1-4050-85b1-1f4b12e178b6","Type":"ContainerStarted","Data":"249b0d432c3a203a8ff151d16b86d1df2dfa0244f122b8665d60cf482ad5212b"}
May 20 15:32:46.425251 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.425192    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-x555j" podStartSLOduration=2.043828894 podStartE2EDuration="4.425176354s" podCreationTimestamp="2026-05-20 15:32:42 +0000 UTC" firstStartedPulling="2026-05-20 15:32:43.312468943 +0000 UTC m=+53.016562514" lastFinishedPulling="2026-05-20 15:32:45.693816385 +0000 UTC m=+55.397909974" observedRunningTime="2026-05-20 15:32:46.423747486 +0000 UTC m=+56.127841081" watchObservedRunningTime="2026-05-20 15:32:46.425176354 +0000 UTC m=+56.129269940"
May 20 15:32:46.467321 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.467250    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2gp62" podStartSLOduration=2.057132218 podStartE2EDuration="4.467231544s" podCreationTimestamp="2026-05-20 15:32:42 +0000 UTC" firstStartedPulling="2026-05-20 15:32:43.287959081 +0000 UTC m=+52.992052649" lastFinishedPulling="2026-05-20 15:32:45.698058405 +0000 UTC m=+55.402151975" observedRunningTime="2026-05-20 15:32:46.446311191 +0000 UTC m=+56.150404781" watchObservedRunningTime="2026-05-20 15:32:46.467231544 +0000 UTC m=+56.171325146"
May 20 15:32:46.467659 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.467548    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"]
May 20 15:32:46.481521 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.481492    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.485330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.483970    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"]
May 20 15:32:46.485501 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.485480    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 20 15:32:46.485578 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.485537    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 20 15:32:46.486224 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.485770    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 20 15:32:46.486224 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.485813    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 20 15:32:46.486224 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.485771    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 20 15:32:46.486224 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.486073    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mv6x7\""
May 20 15:32:46.647153 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.647064    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0cc31f6-0dab-44f5-8917-e52473c3da68-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.647153 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.647106    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0cc31f6-0dab-44f5-8917-e52473c3da68-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.647153 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.647136    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f6rl\" (UniqueName: \"kubernetes.io/projected/c0cc31f6-0dab-44f5-8917-e52473c3da68-kube-api-access-5f6rl\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.647352 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.647262    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0cc31f6-0dab-44f5-8917-e52473c3da68-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.662652 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.662612    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-xvk6q"]
May 20 15:32:46.674636 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.674578    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-xvk6q"
May 20 15:32:46.677663 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.677643    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 20 15:32:46.677663 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.677649    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 20 15:32:46.677826 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.677702    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5lvlx\""
May 20 15:32:46.682266 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.682232    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-xvk6q"]
May 20 15:32:46.748273 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.748229    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0cc31f6-0dab-44f5-8917-e52473c3da68-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.748453 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.748296    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0cc31f6-0dab-44f5-8917-e52473c3da68-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.748453 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.748334    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0cc31f6-0dab-44f5-8917-e52473c3da68-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.748453 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.748365    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5f6rl\" (UniqueName: \"kubernetes.io/projected/c0cc31f6-0dab-44f5-8917-e52473c3da68-kube-api-access-5f6rl\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.748453 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.748420    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ljz\" (UniqueName: \"kubernetes.io/projected/0316ba6c-6a22-4aca-af00-0a93ecf7a6e3-kube-api-access-f9ljz\") pod \"downloads-57bd96b888-xvk6q\" (UID: \"0316ba6c-6a22-4aca-af00-0a93ecf7a6e3\") " pod="openshift-console/downloads-57bd96b888-xvk6q"
May 20 15:32:46.749405 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.749373    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0cc31f6-0dab-44f5-8917-e52473c3da68-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.751070 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.751046    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0cc31f6-0dab-44f5-8917-e52473c3da68-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.751185 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.751107    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0cc31f6-0dab-44f5-8917-e52473c3da68-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.758994 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.758969    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f6rl\" (UniqueName: \"kubernetes.io/projected/c0cc31f6-0dab-44f5-8917-e52473c3da68-kube-api-access-5f6rl\") pod \"prometheus-operator-7779bbdd78-l4vwx\" (UID: \"c0cc31f6-0dab-44f5-8917-e52473c3da68\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.794908 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.794865    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"
May 20 15:32:46.848884 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.848832    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ljz\" (UniqueName: \"kubernetes.io/projected/0316ba6c-6a22-4aca-af00-0a93ecf7a6e3-kube-api-access-f9ljz\") pod \"downloads-57bd96b888-xvk6q\" (UID: \"0316ba6c-6a22-4aca-af00-0a93ecf7a6e3\") " pod="openshift-console/downloads-57bd96b888-xvk6q"
May 20 15:32:46.858329 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.858286    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ljz\" (UniqueName: \"kubernetes.io/projected/0316ba6c-6a22-4aca-af00-0a93ecf7a6e3-kube-api-access-f9ljz\") pod \"downloads-57bd96b888-xvk6q\" (UID: \"0316ba6c-6a22-4aca-af00-0a93ecf7a6e3\") " pod="openshift-console/downloads-57bd96b888-xvk6q"
May 20 15:32:46.984994 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:46.984901    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-xvk6q"
May 20 15:32:47.269141 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:47.269099    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-xvk6q"]
May 20 15:32:47.273855 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:47.273812    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0316ba6c_6a22_4aca_af00_0a93ecf7a6e3.slice/crio-783db6ad52c23f585d03288f456b71dda9110eaa7aacdf40dd12ffa1238c1deb WatchSource:0}: Error finding container 783db6ad52c23f585d03288f456b71dda9110eaa7aacdf40dd12ffa1238c1deb: Status 404 returned error can't find the container with id 783db6ad52c23f585d03288f456b71dda9110eaa7aacdf40dd12ffa1238c1deb
May 20 15:32:47.283969 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:47.283915    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx"]
May 20 15:32:47.385461 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:47.385425    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-xvk6q" event={"ID":"0316ba6c-6a22-4aca-af00-0a93ecf7a6e3","Type":"ContainerStarted","Data":"783db6ad52c23f585d03288f456b71dda9110eaa7aacdf40dd12ffa1238c1deb"}
May 20 15:32:47.403475 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:47.403441    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cc31f6_0dab_44f5_8917_e52473c3da68.slice/crio-317156f48b614e9805f0c99762ff5fb1706e27836de035fc8815c1876e856be4 WatchSource:0}: Error finding container 317156f48b614e9805f0c99762ff5fb1706e27836de035fc8815c1876e856be4: Status 404 returned error can't find the container with id 317156f48b614e9805f0c99762ff5fb1706e27836de035fc8815c1876e856be4
May 20 15:32:48.390356 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:48.390321    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx" event={"ID":"c0cc31f6-0dab-44f5-8917-e52473c3da68","Type":"ContainerStarted","Data":"317156f48b614e9805f0c99762ff5fb1706e27836de035fc8815c1876e856be4"}
May 20 15:32:48.393036 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:48.392997    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p574b" event={"ID":"8d481c73-56b1-4d54-97e3-9344490303ee","Type":"ContainerStarted","Data":"09cd6d079cd3031fd157283bb566d90161aa52aa7a76f0bc88d2bd82157ef450"}
May 20 15:32:48.412967 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:48.412910    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p574b" podStartSLOduration=2.455396581 podStartE2EDuration="6.4128717s" podCreationTimestamp="2026-05-20 15:32:42 +0000 UTC" firstStartedPulling="2026-05-20 15:32:43.463295882 +0000 UTC m=+53.167389463" lastFinishedPulling="2026-05-20 15:32:47.420771013 +0000 UTC m=+57.124864582" observedRunningTime="2026-05-20 15:32:48.411388031 +0000 UTC m=+58.115481635" watchObservedRunningTime="2026-05-20 15:32:48.4128717 +0000 UTC m=+58.116965291"
May 20 15:32:49.398298 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:49.397787    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx" event={"ID":"c0cc31f6-0dab-44f5-8917-e52473c3da68","Type":"ContainerStarted","Data":"41a38032c365bf91fa1692c6d258c06cbd8f356bb5ecb0e485d319318433f132"}
May 20 15:32:49.398298 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:49.397833    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx" event={"ID":"c0cc31f6-0dab-44f5-8917-e52473c3da68","Type":"ContainerStarted","Data":"c3a6c225aa6beec3ed08364a69f18bcc1bf2ac4c93194cd6dc1bf14b48f9cf84"}
May 20 15:32:49.417016 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:49.416963    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-l4vwx" podStartSLOduration=1.830727151 podStartE2EDuration="3.416945919s" podCreationTimestamp="2026-05-20 15:32:46 +0000 UTC" firstStartedPulling="2026-05-20 15:32:47.416498698 +0000 UTC m=+57.120592265" lastFinishedPulling="2026-05-20 15:32:49.002717463 +0000 UTC m=+58.706811033" observedRunningTime="2026-05-20 15:32:49.414931241 +0000 UTC m=+59.119024830" watchObservedRunningTime="2026-05-20 15:32:49.416945919 +0000 UTC m=+59.121039509"
May 20 15:32:50.821440 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.821398    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"]
May 20 15:32:50.824600 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.824565    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:50.827720 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.827691    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 20 15:32:50.827872 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.827691    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 20 15:32:50.827872 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.827696    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bj9lw\""
May 20 15:32:50.831042 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.830983    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"]
May 20 15:32:50.833320 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.833297    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:50.833655 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.833633    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"]
May 20 15:32:50.836155 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.836134    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-whz2g\""
May 20 15:32:50.837565 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.837548    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 20 15:32:50.837741 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.837720    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 20 15:32:50.837835 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.837785    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 20 15:32:50.848551 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.848521    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"]
May 20 15:32:50.864656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.864616    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cpr9c"]
May 20 15:32:50.867186 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.867166    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:50.869959 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.869927    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 20 15:32:50.870102 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.870028    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 20 15:32:50.870102 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.869932    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 20 15:32:50.870221 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.870213    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n8vs4\""
May 20 15:32:50.980393 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980293    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:50.980393 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/30991255-505c-4f6a-8afd-3d25afc9b842-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:50.980673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/507701a3-3d79-4d09-ba31-8e27a16b7edf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:50.980673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980464    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-wtmp\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:50.980673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980491    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a0e960d-021a-4d9c-921a-7c3f884015d1-root\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:50.980673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a0e960d-021a-4d9c-921a-7c3f884015d1-metrics-client-ca\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:50.980673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980542    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfb8d\" (UniqueName: \"kubernetes.io/projected/5a0e960d-021a-4d9c-921a-7c3f884015d1-kube-api-access-kfb8d\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:50.980673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980567    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a0e960d-021a-4d9c-921a-7c3f884015d1-sys\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:50.980673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980629    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-textfile\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:50.980673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980655    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-accelerators-collector-config\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:50.981049 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980686    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/507701a3-3d79-4d09-ba31-8e27a16b7edf-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:50.981049 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980744    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30991255-505c-4f6a-8afd-3d25afc9b842-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:50.981049 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980772    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:50.981049 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980800    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44mc\" (UniqueName: \"kubernetes.io/projected/507701a3-3d79-4d09-ba31-8e27a16b7edf-kube-api-access-q44mc\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:50.981049 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980826    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/507701a3-3d79-4d09-ba31-8e27a16b7edf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:50.981049 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980853    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-tls\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:50.981049 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980883    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:50.981049 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980915    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:50.981049 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:50.980940    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmcwv\" (UniqueName: \"kubernetes.io/projected/30991255-505c-4f6a-8afd-3d25afc9b842-kube-api-access-fmcwv\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.082051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.081959    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/507701a3-3d79-4d09-ba31-8e27a16b7edf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:51.082051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.082008    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-tls\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.082261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.082121    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.082261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.082161    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.082261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.082194    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fmcwv\" (UniqueName: \"kubernetes.io/projected/30991255-505c-4f6a-8afd-3d25afc9b842-kube-api-access-fmcwv\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.082261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.082251    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.082478 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.082293    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/30991255-505c-4f6a-8afd-3d25afc9b842-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.082478 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.082344    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/507701a3-3d79-4d09-ba31-8e27a16b7edf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:51.083011 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.082969    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-wtmp\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.083154 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083015    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a0e960d-021a-4d9c-921a-7c3f884015d1-root\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.083154 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083041    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a0e960d-021a-4d9c-921a-7c3f884015d1-metrics-client-ca\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.083154 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083069    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kfb8d\" (UniqueName: \"kubernetes.io/projected/5a0e960d-021a-4d9c-921a-7c3f884015d1-kube-api-access-kfb8d\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.083154 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083098    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a0e960d-021a-4d9c-921a-7c3f884015d1-sys\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.083154 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083103    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a0e960d-021a-4d9c-921a-7c3f884015d1-root\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.083154 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083104    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-wtmp\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.083455 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083170    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-textfile\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.083455 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083178    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a0e960d-021a-4d9c-921a-7c3f884015d1-sys\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.083455 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083212    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-accelerators-collector-config\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.083455 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083259    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/507701a3-3d79-4d09-ba31-8e27a16b7edf-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:51.083455 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083319    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30991255-505c-4f6a-8afd-3d25afc9b842-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.083455 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083348    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.083455 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083373    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q44mc\" (UniqueName: \"kubernetes.io/projected/507701a3-3d79-4d09-ba31-8e27a16b7edf-kube-api-access-q44mc\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:51.084219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083553    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-textfile\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.084219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083749    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/30991255-505c-4f6a-8afd-3d25afc9b842-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.084219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.083798    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a0e960d-021a-4d9c-921a-7c3f884015d1-metrics-client-ca\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.084219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.084012    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/507701a3-3d79-4d09-ba31-8e27a16b7edf-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:51.084219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.084107    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30991255-505c-4f6a-8afd-3d25afc9b842-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.084835 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.084815    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 20 15:32:51.084945 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.084870    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 20 15:32:51.084945 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.084887    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 20 15:32:51.086073 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.086044    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 20 15:32:51.086165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.086107    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 20 15:32:51.086226 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.086179    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 20 15:32:51.087338 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.087109    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 20 15:32:51.089837 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.088569    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 20 15:32:51.092886 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:51.092854    2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 20 15:32:51.092980 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:51.092932    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/507701a3-3d79-4d09-ba31-8e27a16b7edf-openshift-state-metrics-tls podName:507701a3-3d79-4d09-ba31-8e27a16b7edf nodeName:}" failed. No retries permitted until 2026-05-20 15:32:51.592909082 +0000 UTC m=+61.297002671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/507701a3-3d79-4d09-ba31-8e27a16b7edf-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-r2m4h" (UID: "507701a3-3d79-4d09-ba31-8e27a16b7edf") : secret "openshift-state-metrics-tls" not found
May 20 15:32:51.093103 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:51.093082    2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 20 15:32:51.093186 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:51.093162    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-tls podName:5a0e960d-021a-4d9c-921a-7c3f884015d1 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:51.593143573 +0000 UTC m=+61.297237157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-tls") pod "node-exporter-cpr9c" (UID: "5a0e960d-021a-4d9c-921a-7c3f884015d1") : secret "node-exporter-tls" not found
May 20 15:32:51.093506 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:51.093401    2567 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
May 20 15:32:51.093506 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:51.093469    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-tls podName:30991255-505c-4f6a-8afd-3d25afc9b842 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:51.593453295 +0000 UTC m=+61.297546881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-77qrl" (UID: "30991255-505c-4f6a-8afd-3d25afc9b842") : secret "kube-state-metrics-tls" not found
May 20 15:32:51.093863 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.093688    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44mc\" (UniqueName: \"kubernetes.io/projected/507701a3-3d79-4d09-ba31-8e27a16b7edf-kube-api-access-q44mc\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:51.094523 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.094118    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-accelerators-collector-config\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.094523 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.094198    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.094523 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.094480    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmcwv\" (UniqueName: \"kubernetes.io/projected/30991255-505c-4f6a-8afd-3d25afc9b842-kube-api-access-fmcwv\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.095256 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.095207    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfb8d\" (UniqueName: \"kubernetes.io/projected/5a0e960d-021a-4d9c-921a-7c3f884015d1-kube-api-access-kfb8d\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.095872 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.095847    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/507701a3-3d79-4d09-ba31-8e27a16b7edf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:51.095950 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.095908    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.097067 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.097046    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.688469 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.688429    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.688746 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.688569    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/507701a3-3d79-4d09-ba31-8e27a16b7edf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:51.688746 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.688621    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-tls\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.691306 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.691274    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a0e960d-021a-4d9c-921a-7c3f884015d1-node-exporter-tls\") pod \"node-exporter-cpr9c\" (UID: \"5a0e960d-021a-4d9c-921a-7c3f884015d1\") " pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.691306 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.691292    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/507701a3-3d79-4d09-ba31-8e27a16b7edf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-r2m4h\" (UID: \"507701a3-3d79-4d09-ba31-8e27a16b7edf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:51.691461 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.691340    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/30991255-505c-4f6a-8afd-3d25afc9b842-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-77qrl\" (UID: \"30991255-505c-4f6a-8afd-3d25afc9b842\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.741066 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.741030    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bj9lw\""
May 20 15:32:51.748624 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.748567    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"
May 20 15:32:51.748800 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.748743    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-whz2g\""
May 20 15:32:51.757083 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.756998    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"
May 20 15:32:51.781505 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.781287    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n8vs4\""
May 20 15:32:51.789643 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.789609    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cpr9c"
May 20 15:32:51.802072 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:51.802040    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0e960d_021a_4d9c_921a_7c3f884015d1.slice/crio-9d9cdc4cb2b99b38cdfd2369b7f929a86fba9cf2f8fabfc583be0c20fb416f84 WatchSource:0}: Error finding container 9d9cdc4cb2b99b38cdfd2369b7f929a86fba9cf2f8fabfc583be0c20fb416f84: Status 404 returned error can't find the container with id 9d9cdc4cb2b99b38cdfd2369b7f929a86fba9cf2f8fabfc583be0c20fb416f84
May 20 15:32:51.922966 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.922932    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:32:51.926812 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.926786    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h"]
May 20 15:32:51.927005 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.926986    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:51.930288 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:51.930263    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507701a3_3d79_4d09_ba31_8e27a16b7edf.slice/crio-7c85ce28ca667153694ff8e8f666a8767137f90b5827e709fe2a72d9331f1d98 WatchSource:0}: Error finding container 7c85ce28ca667153694ff8e8f666a8767137f90b5827e709fe2a72d9331f1d98: Status 404 returned error can't find the container with id 7c85ce28ca667153694ff8e8f666a8767137f90b5827e709fe2a72d9331f1d98
May 20 15:32:51.930686 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.930668    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 20 15:32:51.930884 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.930856    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 20 15:32:51.930884 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.930870    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 20 15:32:51.931163 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.931136    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 20 15:32:51.931295 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.931277    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 20 15:32:51.931419 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.931403    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 20 15:32:51.931717 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.931696    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 20 15:32:51.932012 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.931913    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 20 15:32:51.932333 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.932315    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gccw9\""
May 20 15:32:51.932922 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.932900    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 20 15:32:51.943936 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.943360    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl"]
May 20 15:32:51.944807 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:51.944783    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:32:51.948289 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:51.948256    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30991255_505c_4f6a_8afd_3d25afc9b842.slice/crio-007f29b3ccbc418da9b9d7a717ddabdb263e30e772b9c052dae0c2a8231548fa WatchSource:0}: Error finding container 007f29b3ccbc418da9b9d7a717ddabdb263e30e772b9c052dae0c2a8231548fa: Status 404 returned error can't find the container with id 007f29b3ccbc418da9b9d7a717ddabdb263e30e772b9c052dae0c2a8231548fa
May 20 15:32:52.092361 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092329    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-volume\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.092361 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092365    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.092540 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092384    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-web-config\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.092540 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.092540 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092424    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.092540 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092453    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vndv6\" (UniqueName: \"kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-kube-api-access-vndv6\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.092540 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092520    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.092742 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092567    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-out\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.092798 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092784    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.092853 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092824    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.092904 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092856    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.092904 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092880    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.093018 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.092997    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194046 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.193956    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194046 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194016    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-volume\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194046 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194035    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194054    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-web-config\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194076    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194094    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194111    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vndv6\" (UniqueName: \"kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-kube-api-access-vndv6\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194135    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194157    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-out\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194191    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194222    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194252    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.194311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.194275    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.196712 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:52.195514    2567 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 20 15:32:52.196712 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:52.195609    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-main-tls podName:d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6 nodeName:}" failed. No retries permitted until 2026-05-20 15:32:52.695563563 +0000 UTC m=+62.399657130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6") : secret "alertmanager-main-tls" not found
May 20 15:32:52.196712 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.195736    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.197248 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.197223    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.197469 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.197399    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.200294 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.200268    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.200777 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.200755    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.201249 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.201224    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.201975 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.201620    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-out\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.201975 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.201716    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-web-config\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.201975 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.201931    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.202937 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.202914    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.203721 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.203698    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-volume\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.212372 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.212349    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vndv6\" (UniqueName: \"kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-kube-api-access-vndv6\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.411011 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.410966    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h" event={"ID":"507701a3-3d79-4d09-ba31-8e27a16b7edf","Type":"ContainerStarted","Data":"84108e4942c9b79d4bec2959dc2fdc74f982cd9a03f8a4a6000d826d931dcb3d"}
May 20 15:32:52.411011 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.411015    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h" event={"ID":"507701a3-3d79-4d09-ba31-8e27a16b7edf","Type":"ContainerStarted","Data":"c8a18dc6e5fd6ddb857009680487b6df9e79c793a0cca984f12cdbf677eb0083"}
May 20 15:32:52.411242 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.411030    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h" event={"ID":"507701a3-3d79-4d09-ba31-8e27a16b7edf","Type":"ContainerStarted","Data":"7c85ce28ca667153694ff8e8f666a8767137f90b5827e709fe2a72d9331f1d98"}
May 20 15:32:52.412503 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.412466    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cpr9c" event={"ID":"5a0e960d-021a-4d9c-921a-7c3f884015d1","Type":"ContainerStarted","Data":"9d9cdc4cb2b99b38cdfd2369b7f929a86fba9cf2f8fabfc583be0c20fb416f84"}
May 20 15:32:52.413760 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.413725    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl" event={"ID":"30991255-505c-4f6a-8afd-3d25afc9b842","Type":"ContainerStarted","Data":"007f29b3ccbc418da9b9d7a717ddabdb263e30e772b9c052dae0c2a8231548fa"}
May 20 15:32:52.699702 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.699657    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.711465 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.711427    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:52.837693 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:52.837657    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 15:32:53.023214 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.023185    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:32:53.381699 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:53.381638    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f7ece4_fb90_4a62_8ff1_b1f0136d8be6.slice/crio-861b9590b1ba909d5f8b228cbf635cd25d6aefd5614b8ef31fc33ebd76c95e5b WatchSource:0}: Error finding container 861b9590b1ba909d5f8b228cbf635cd25d6aefd5614b8ef31fc33ebd76c95e5b: Status 404 returned error can't find the container with id 861b9590b1ba909d5f8b228cbf635cd25d6aefd5614b8ef31fc33ebd76c95e5b
May 20 15:32:53.420179 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.420139    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerStarted","Data":"861b9590b1ba909d5f8b228cbf635cd25d6aefd5614b8ef31fc33ebd76c95e5b"}
May 20 15:32:53.422331 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.422302    2567 generic.go:358] "Generic (PLEG): container finished" podID="5a0e960d-021a-4d9c-921a-7c3f884015d1" containerID="eb71e7f3405eef8f9e181dabf4790d02d0240e98211e3fd2a6c676f6931f59ad" exitCode=0
May 20 15:32:53.422495 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.422363    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cpr9c" event={"ID":"5a0e960d-021a-4d9c-921a-7c3f884015d1","Type":"ContainerDied","Data":"eb71e7f3405eef8f9e181dabf4790d02d0240e98211e3fd2a6c676f6931f59ad"}
May 20 15:32:53.920608 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.920556    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"]
May 20 15:32:53.924626 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.924506    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:53.927614 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.927526    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 20 15:32:53.927731 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.927612    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 20 15:32:53.928115 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.927859    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 20 15:32:53.928115 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.927934    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-95aadm1orvtjo\""
May 20 15:32:53.928115 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.927859    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s42bd\""
May 20 15:32:53.928115 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.928054    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 20 15:32:53.928408 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.928293    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 20 15:32:53.937519 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:53.937229    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"]
May 20 15:32:54.113022 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.112992    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2rf\" (UniqueName: \"kubernetes.io/projected/fc05d954-00b2-4086-a171-f90af0bdaf0f-kube-api-access-wp2rf\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.113514 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.113037    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.113514 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.113065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.113514 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.113109    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.113514 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.113149    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.113514 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.113213    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-tls\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.113514 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.113271    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc05d954-00b2-4086-a171-f90af0bdaf0f-metrics-client-ca\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.113514 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.113314    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-grpc-tls\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.214260 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.214168    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc05d954-00b2-4086-a171-f90af0bdaf0f-metrics-client-ca\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.214260 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.214227    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-grpc-tls\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.214488 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.214271    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2rf\" (UniqueName: \"kubernetes.io/projected/fc05d954-00b2-4086-a171-f90af0bdaf0f-kube-api-access-wp2rf\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.214488 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.214300    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.214488 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.214326    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.214488 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.214377    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.214488 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.214412    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.214753 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.214508    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-tls\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.217542 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.217457    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc05d954-00b2-4086-a171-f90af0bdaf0f-metrics-client-ca\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.220301 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.220130    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.220673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.220568    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.221494 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.221449    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.222668 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.222625    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-grpc-tls\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.224527 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.223736    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.224527 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.224023    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc05d954-00b2-4086-a171-f90af0bdaf0f-secret-thanos-querier-tls\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.243945 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.243860    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2rf\" (UniqueName: \"kubernetes.io/projected/fc05d954-00b2-4086-a171-f90af0bdaf0f-kube-api-access-wp2rf\") pod \"thanos-querier-6d796ddbc9-bbf6k\" (UID: \"fc05d954-00b2-4086-a171-f90af0bdaf0f\") " pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.244337 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.244309    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:32:54.403352 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.401804    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"]
May 20 15:32:54.428212 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.428172    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h" event={"ID":"507701a3-3d79-4d09-ba31-8e27a16b7edf","Type":"ContainerStarted","Data":"af3e9949f5087ecd525b57931e140f88a971ba301e3d12270cec5f36ae4cb555"}
May 20 15:32:54.431670 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.431644    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cpr9c" event={"ID":"5a0e960d-021a-4d9c-921a-7c3f884015d1","Type":"ContainerStarted","Data":"2d00f1e3e6f7e4f1c6e6979b6073c1fc7ac00f0e8a1ff57f882dbc636d2ba2d3"}
May 20 15:32:54.431670 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.431676    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cpr9c" event={"ID":"5a0e960d-021a-4d9c-921a-7c3f884015d1","Type":"ContainerStarted","Data":"4932abad9b3aa5db43db8992566efd4b37c0532595479e745283a4cafa34a212"}
May 20 15:32:54.434243 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.434218    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl" event={"ID":"30991255-505c-4f6a-8afd-3d25afc9b842","Type":"ContainerStarted","Data":"c4c6944c1117865d7a6035ee418b61e89117126191d2e31f11d3edd53998bce1"}
May 20 15:32:54.434369 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.434251    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl" event={"ID":"30991255-505c-4f6a-8afd-3d25afc9b842","Type":"ContainerStarted","Data":"c43194fc95270477b6fa4ca54d9b97b6e1b4a2c427d85bad0b7d851a54c934d5"}
May 20 15:32:54.434369 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.434265    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl" event={"ID":"30991255-505c-4f6a-8afd-3d25afc9b842","Type":"ContainerStarted","Data":"61291883886c2eb82318aa9f0ba215adf4afd02375bb397eb52ff5645308e428"}
May 20 15:32:54.450908 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.450849    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-r2m4h" podStartSLOduration=2.850420316 podStartE2EDuration="4.450830087s" podCreationTimestamp="2026-05-20 15:32:50 +0000 UTC" firstStartedPulling="2026-05-20 15:32:52.067088083 +0000 UTC m=+61.771181650" lastFinishedPulling="2026-05-20 15:32:53.667497852 +0000 UTC m=+63.371591421" observedRunningTime="2026-05-20 15:32:54.45026587 +0000 UTC m=+64.154359461" watchObservedRunningTime="2026-05-20 15:32:54.450830087 +0000 UTC m=+64.154923680"
May 20 15:32:54.485634 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.485506    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-77qrl" podStartSLOduration=2.771494912 podStartE2EDuration="4.485494649s" podCreationTimestamp="2026-05-20 15:32:50 +0000 UTC" firstStartedPulling="2026-05-20 15:32:51.952508092 +0000 UTC m=+61.656601663" lastFinishedPulling="2026-05-20 15:32:53.666507824 +0000 UTC m=+63.370601400" observedRunningTime="2026-05-20 15:32:54.483249699 +0000 UTC m=+64.187343319" watchObservedRunningTime="2026-05-20 15:32:54.485494649 +0000 UTC m=+64.189588239"
May 20 15:32:54.505855 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.505793    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cpr9c" podStartSLOduration=3.500607794 podStartE2EDuration="4.505773775s" podCreationTimestamp="2026-05-20 15:32:50 +0000 UTC" firstStartedPulling="2026-05-20 15:32:51.804479875 +0000 UTC m=+61.508573448" lastFinishedPulling="2026-05-20 15:32:52.809645861 +0000 UTC m=+62.513739429" observedRunningTime="2026-05-20 15:32:54.504703941 +0000 UTC m=+64.208797531" watchObservedRunningTime="2026-05-20 15:32:54.505773775 +0000 UTC m=+64.209867368"
May 20 15:32:54.726013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:54.725975    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc05d954_00b2_4086_a171_f90af0bdaf0f.slice/crio-22893e445a9c898f359449ea6bde7e672173a44f7ee27e69b2e3aeb8da70bee7 WatchSource:0}: Error finding container 22893e445a9c898f359449ea6bde7e672173a44f7ee27e69b2e3aeb8da70bee7: Status 404 returned error can't find the container with id 22893e445a9c898f359449ea6bde7e672173a44f7ee27e69b2e3aeb8da70bee7
May 20 15:32:54.779083 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.779008    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-65cbc4f8b8-6jzs8"]
May 20 15:32:54.800042 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.800008    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cbc4f8b8-6jzs8"]
May 20 15:32:54.800187 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.800148    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:54.803195 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.803005    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 20 15:32:54.803195 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.803064    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xnnld\""
May 20 15:32:54.803195 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.803117    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 20 15:32:54.803195 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.803179    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 20 15:32:54.803472 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.803455    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 20 15:32:54.803518 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.803479    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 20 15:32:54.922192 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.922153    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-serving-cert\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:54.922352 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.922249    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-config\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:54.922352 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.922301    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-oauth-config\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:54.922352 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.922326    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-service-ca\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:54.922501 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.922357    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpl5\" (UniqueName: \"kubernetes.io/projected/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-kube-api-access-kbpl5\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:54.922561 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:54.922498    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-oauth-serving-cert\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.023618 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.023556    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-serving-cert\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.023809 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.023645    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-config\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.023809 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.023694    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-oauth-config\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.023809 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.023719    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-service-ca\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.023809 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.023747    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpl5\" (UniqueName: \"kubernetes.io/projected/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-kube-api-access-kbpl5\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.024021 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.023827    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-oauth-serving-cert\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.024627 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.024458    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-service-ca\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.024627 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.024542    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-config\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.024627 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.024560    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-oauth-serving-cert\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.026636 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.026613    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-serving-cert\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.026742 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.026697    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-oauth-config\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.041453 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.041385    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpl5\" (UniqueName: \"kubernetes.io/projected/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-kube-api-access-kbpl5\") pod \"console-65cbc4f8b8-6jzs8\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") " pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.112231 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.112189    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:32:55.126124 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.126093    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"]
May 20 15:32:55.152035 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.152005    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"]
May 20 15:32:55.152167 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.152141    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.155417 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.155384    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 20 15:32:55.155693 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.155666    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 20 15:32:55.156103 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.155898    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 20 15:32:55.156601 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.156559    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 20 15:32:55.156864 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.156844    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c28hnq97egevt\""
May 20 15:32:55.157277 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.157058    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ntb57\""
May 20 15:32:55.271245 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.271213    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65cbc4f8b8-6jzs8"]
May 20 15:32:55.316636 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:55.316542    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8e2758_cd74_40bc_851f_f74c33c0d5a6.slice/crio-ad75975a781cf103c25d13f11d1fb5ad1d9712f951a21f3a184914633425cd01 WatchSource:0}: Error finding container ad75975a781cf103c25d13f11d1fb5ad1d9712f951a21f3a184914633425cd01: Status 404 returned error can't find the container with id ad75975a781cf103c25d13f11d1fb5ad1d9712f951a21f3a184914633425cd01
May 20 15:32:55.327666 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.327643    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dthzq\" (UniqueName: \"kubernetes.io/projected/a46d6274-fc02-4bea-8ba5-31040d263711-kube-api-access-dthzq\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.327780 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.327727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a46d6274-fc02-4bea-8ba5-31040d263711-audit-log\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.327844 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.327788    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a46d6274-fc02-4bea-8ba5-31040d263711-secret-metrics-server-tls\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.327891 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.327860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a46d6274-fc02-4bea-8ba5-31040d263711-secret-metrics-server-client-certs\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.327937 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.327921    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a46d6274-fc02-4bea-8ba5-31040d263711-metrics-server-audit-profiles\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.327982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.327970    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a46d6274-fc02-4bea-8ba5-31040d263711-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.328020 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.327997    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a46d6274-fc02-4bea-8ba5-31040d263711-client-ca-bundle\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.428417 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.428373    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a46d6274-fc02-4bea-8ba5-31040d263711-client-ca-bundle\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.428578 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.428429    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dthzq\" (UniqueName: \"kubernetes.io/projected/a46d6274-fc02-4bea-8ba5-31040d263711-kube-api-access-dthzq\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.428578 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.428516    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a46d6274-fc02-4bea-8ba5-31040d263711-audit-log\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.428578 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.428557    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a46d6274-fc02-4bea-8ba5-31040d263711-secret-metrics-server-tls\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.428810 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.428778    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a46d6274-fc02-4bea-8ba5-31040d263711-secret-metrics-server-client-certs\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.428935 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.428846    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a46d6274-fc02-4bea-8ba5-31040d263711-metrics-server-audit-profiles\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.428935 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.428892    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a46d6274-fc02-4bea-8ba5-31040d263711-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.429227 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.429193    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a46d6274-fc02-4bea-8ba5-31040d263711-audit-log\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.429913 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.429887    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a46d6274-fc02-4bea-8ba5-31040d263711-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.430277 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.430254    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a46d6274-fc02-4bea-8ba5-31040d263711-metrics-server-audit-profiles\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.432207 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.432183    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a46d6274-fc02-4bea-8ba5-31040d263711-secret-metrics-server-tls\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.432302 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.432217    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a46d6274-fc02-4bea-8ba5-31040d263711-secret-metrics-server-client-certs\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.432302 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.432259    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a46d6274-fc02-4bea-8ba5-31040d263711-client-ca-bundle\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.438723 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.438701    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dthzq\" (UniqueName: \"kubernetes.io/projected/a46d6274-fc02-4bea-8ba5-31040d263711-kube-api-access-dthzq\") pod \"metrics-server-6d685b6d6c-pn6kp\" (UID: \"a46d6274-fc02-4bea-8ba5-31040d263711\") " pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.440495 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.440467    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cbc4f8b8-6jzs8" event={"ID":"7c8e2758-cd74-40bc-851f-f74c33c0d5a6","Type":"ContainerStarted","Data":"ad75975a781cf103c25d13f11d1fb5ad1d9712f951a21f3a184914633425cd01"}
May 20 15:32:55.442768 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.442663    2567 generic.go:358] "Generic (PLEG): container finished" podID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerID="e38d9a9976271eb37738d0818da5f2ce0acbdd211f19d079eabb4318cf450a10" exitCode=0
May 20 15:32:55.442768 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.442735    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerDied","Data":"e38d9a9976271eb37738d0818da5f2ce0acbdd211f19d079eabb4318cf450a10"}
May 20 15:32:55.444528 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.444492    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k" event={"ID":"fc05d954-00b2-4086-a171-f90af0bdaf0f","Type":"ContainerStarted","Data":"22893e445a9c898f359449ea6bde7e672173a44f7ee27e69b2e3aeb8da70bee7"}
May 20 15:32:55.479347 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.478931    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:32:55.614059 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.612641    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-wvntl"]
May 20 15:32:55.645092 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.644956    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-wvntl"]
May 20 15:32:55.645316 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.645153    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-wvntl"
May 20 15:32:55.649044 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.648036    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-tr9l4\""
May 20 15:32:55.649044 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.648294    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 20 15:32:55.650231 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.649925    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"]
May 20 15:32:55.656165 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:32:55.656135    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46d6274_fc02_4bea_8ba5_31040d263711.slice/crio-0545a614e26b97c40d2cb06e9e82c4617149062e4b5ef748d690522cb8fe1f72 WatchSource:0}: Error finding container 0545a614e26b97c40d2cb06e9e82c4617149062e4b5ef748d690522cb8fe1f72: Status 404 returned error can't find the container with id 0545a614e26b97c40d2cb06e9e82c4617149062e4b5ef748d690522cb8fe1f72
May 20 15:32:55.732096 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.732052    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a204308-a018-4037-a3db-8bc7a0e5444c-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-wvntl\" (UID: \"3a204308-a018-4037-a3db-8bc7a0e5444c\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-wvntl"
May 20 15:32:55.833115 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:55.833074    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a204308-a018-4037-a3db-8bc7a0e5444c-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-wvntl\" (UID: \"3a204308-a018-4037-a3db-8bc7a0e5444c\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-wvntl"
May 20 15:32:55.833282 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:55.833186    2567 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 20 15:32:55.833282 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:32:55.833265    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a204308-a018-4037-a3db-8bc7a0e5444c-monitoring-plugin-cert podName:3a204308-a018-4037-a3db-8bc7a0e5444c nodeName:}" failed. No retries permitted until 2026-05-20 15:32:56.333245197 +0000 UTC m=+66.037338769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/3a204308-a018-4037-a3db-8bc7a0e5444c-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-wvntl" (UID: "3a204308-a018-4037-a3db-8bc7a0e5444c") : secret "monitoring-plugin-cert" not found
May 20 15:32:56.049709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.049667    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"]
May 20 15:32:56.069862 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.068729    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.070838 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.070812    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"]
May 20 15:32:56.072391 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.072364    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 20 15:32:56.073029 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.072750    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 20 15:32:56.073612 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.073236    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 20 15:32:56.074156 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.073813    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bc5zx\""
May 20 15:32:56.074156 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.074043    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 20 15:32:56.078379 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.075892    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 20 15:32:56.082424 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.082402    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 20 15:32:56.137900 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.137860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fe0f182-2f5b-41bc-a208-6df929d947da-metrics-client-ca\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.137900 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.137918    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7255\" (UniqueName: \"kubernetes.io/projected/1fe0f182-2f5b-41bc-a208-6df929d947da-kube-api-access-m7255\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.138521 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.137965    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fe0f182-2f5b-41bc-a208-6df929d947da-serving-certs-ca-bundle\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.138521 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.137990    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-secret-telemeter-client\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.138521 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.138085    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fe0f182-2f5b-41bc-a208-6df929d947da-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.138521 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.138115    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.138521 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.138146    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-telemeter-client-tls\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.138521 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.138169    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-federate-client-tls\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.239226 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.239197    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fe0f182-2f5b-41bc-a208-6df929d947da-metrics-client-ca\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.239412 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.239260    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m7255\" (UniqueName: \"kubernetes.io/projected/1fe0f182-2f5b-41bc-a208-6df929d947da-kube-api-access-m7255\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.239412 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.239336    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fe0f182-2f5b-41bc-a208-6df929d947da-serving-certs-ca-bundle\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.239412 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.239367    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-secret-telemeter-client\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.239574 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.239435    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fe0f182-2f5b-41bc-a208-6df929d947da-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.239574 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.239466    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.239574 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.239501    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-telemeter-client-tls\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.239574 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.239526    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-federate-client-tls\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.240510 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.240418    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fe0f182-2f5b-41bc-a208-6df929d947da-serving-certs-ca-bundle\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.240510 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.240433    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fe0f182-2f5b-41bc-a208-6df929d947da-metrics-client-ca\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.241670 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.241505    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fe0f182-2f5b-41bc-a208-6df929d947da-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.246218 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.245756    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.249505 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.249455    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-federate-client-tls\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.250331 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.250026    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-secret-telemeter-client\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.256567 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.256517    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7255\" (UniqueName: \"kubernetes.io/projected/1fe0f182-2f5b-41bc-a208-6df929d947da-kube-api-access-m7255\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.257407 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.257357    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1fe0f182-2f5b-41bc-a208-6df929d947da-telemeter-client-tls\") pod \"telemeter-client-6bc566fb97-8bt9g\" (UID: \"1fe0f182-2f5b-41bc-a208-6df929d947da\") " pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.340649 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.340201    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a204308-a018-4037-a3db-8bc7a0e5444c-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-wvntl\" (UID: \"3a204308-a018-4037-a3db-8bc7a0e5444c\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-wvntl"
May 20 15:32:56.345199 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.345128    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a204308-a018-4037-a3db-8bc7a0e5444c-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-wvntl\" (UID: \"3a204308-a018-4037-a3db-8bc7a0e5444c\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-wvntl"
May 20 15:32:56.389737 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.388788    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x555j"
May 20 15:32:56.393932 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.393901    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"
May 20 15:32:56.455845 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.455810    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp" event={"ID":"a46d6274-fc02-4bea-8ba5-31040d263711","Type":"ContainerStarted","Data":"0545a614e26b97c40d2cb06e9e82c4617149062e4b5ef748d690522cb8fe1f72"}
May 20 15:32:56.558894 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.558856    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-wvntl"
May 20 15:32:56.745232 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.744870    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:56.748577 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.748359    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 20 15:32:56.760163 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.760099    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f8d9d11-6bdb-402e-ab74-7485d6c52a79-metrics-certs\") pod \"network-metrics-daemon-46dsm\" (UID: \"9f8d9d11-6bdb-402e-ab74-7485d6c52a79\") " pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:56.847700 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.847276    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-682mt\" (UniqueName: \"kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt\") pod \"network-check-target-q4gpd\" (UID: \"ddb4cb56-7904-4742-ac26-ecbd4f38df78\") " pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:56.850659 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.850414    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 20 15:32:56.866648 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.866465    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 20 15:32:56.872345 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.872159    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hm9j9\""
May 20 15:32:56.875323 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.875265    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-682mt\" (UniqueName: \"kubernetes.io/projected/ddb4cb56-7904-4742-ac26-ecbd4f38df78-kube-api-access-682mt\") pod \"network-check-target-q4gpd\" (UID: \"ddb4cb56-7904-4742-ac26-ecbd4f38df78\") " pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:32:56.883439 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.882911    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-46dsm"
May 20 15:32:56.973018 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.972754    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9zz48\""
May 20 15:32:56.978574 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:32:56.977975    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:33:01.983289 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:01.983156    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-568c5f6977-frpl4"]
May 20 15:33:01.985468 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:01.985448    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:01.995364 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:01.995331    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-568c5f6977-frpl4"]
May 20 15:33:01.996534 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:01.996509    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 20 15:33:02.114675 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.114637    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj5pn\" (UniqueName: \"kubernetes.io/projected/f80cf51f-e753-4278-8d78-65728dac7bdc-kube-api-access-sj5pn\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.114873 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.114695    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-console-config\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.114873 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.114724    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-oauth-serving-cert\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.114873 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.114753    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-serving-cert\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.115046 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.114887    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-oauth-config\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.115046 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.114937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-service-ca\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.115046 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.114976    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-trusted-ca-bundle\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.215800 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.215753    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sj5pn\" (UniqueName: \"kubernetes.io/projected/f80cf51f-e753-4278-8d78-65728dac7bdc-kube-api-access-sj5pn\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.215964 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.215835    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-console-config\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.215964 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.215891    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-oauth-serving-cert\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.215964 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.215919    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-serving-cert\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.215964 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.215958    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-oauth-config\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.216150 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.215987    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-service-ca\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.216150 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.216021    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-trusted-ca-bundle\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.216630 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.216561    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-console-config\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.216630 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.216613    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-oauth-serving-cert\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.216824 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.216765    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-service-ca\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.216925 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.216908    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-trusted-ca-bundle\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.218940 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.218914    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-oauth-config\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.219104 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.219075    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-serving-cert\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.225687 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.225660    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj5pn\" (UniqueName: \"kubernetes.io/projected/f80cf51f-e753-4278-8d78-65728dac7bdc-kube-api-access-sj5pn\") pod \"console-568c5f6977-frpl4\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") " pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:02.296701 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:02.296578    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:06.421906 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.421772    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-wvntl"]
May 20 15:33:06.434185 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:33:06.433247    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a204308_a018_4037_a3db_8bc7a0e5444c.slice/crio-433c8a85e8e1965ffd7295d8061140b9de43e7cb94d90092ff61e7fa85893b41 WatchSource:0}: Error finding container 433c8a85e8e1965ffd7295d8061140b9de43e7cb94d90092ff61e7fa85893b41: Status 404 returned error can't find the container with id 433c8a85e8e1965ffd7295d8061140b9de43e7cb94d90092ff61e7fa85893b41
May 20 15:33:06.496925 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.496889    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6bc566fb97-8bt9g"]
May 20 15:33:06.497215 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.497188    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-46dsm"]
May 20 15:33:06.507903 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.506282    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cbc4f8b8-6jzs8" event={"ID":"7c8e2758-cd74-40bc-851f-f74c33c0d5a6","Type":"ContainerStarted","Data":"315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa"}
May 20 15:33:06.521192 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.520359    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-xvk6q" event={"ID":"0316ba6c-6a22-4aca-af00-0a93ecf7a6e3","Type":"ContainerStarted","Data":"2c15f40fe355764ede803fafcedff654b8a275c8d2de7c4b0f5822b47fcf6bf1"}
May 20 15:33:06.521192 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.521147    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-xvk6q"
May 20 15:33:06.522568 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.522506    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g" event={"ID":"1fe0f182-2f5b-41bc-a208-6df929d947da","Type":"ContainerStarted","Data":"339fc5d2259d2975540ff930a3754c9fc2ee3a56d3bdb770fe1343a0aaf30717"}
May 20 15:33:06.526664 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.526547    2567 patch_prober.go:28] interesting pod/downloads-57bd96b888-xvk6q container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.9:8080/\": dial tcp 10.134.0.9:8080: connect: connection refused" start-of-body=
May 20 15:33:06.527135 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.526841    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-57bd96b888-xvk6q" podUID="0316ba6c-6a22-4aca-af00-0a93ecf7a6e3" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.9:8080/\": dial tcp 10.134.0.9:8080: connect: connection refused"
May 20 15:33:06.529553 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.529393    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-wvntl" event={"ID":"3a204308-a018-4037-a3db-8bc7a0e5444c","Type":"ContainerStarted","Data":"433c8a85e8e1965ffd7295d8061140b9de43e7cb94d90092ff61e7fa85893b41"}
May 20 15:33:06.534543 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.533045    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-65cbc4f8b8-6jzs8" podStartSLOduration=1.611066304 podStartE2EDuration="12.533028372s" podCreationTimestamp="2026-05-20 15:32:54 +0000 UTC" firstStartedPulling="2026-05-20 15:32:55.318693582 +0000 UTC m=+65.022787153" lastFinishedPulling="2026-05-20 15:33:06.240655637 +0000 UTC m=+75.944749221" observedRunningTime="2026-05-20 15:33:06.526806394 +0000 UTC m=+76.230899985" watchObservedRunningTime="2026-05-20 15:33:06.533028372 +0000 UTC m=+76.237121965"
May 20 15:33:06.535077 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.535055    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q4gpd"]
May 20 15:33:06.546389 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.546364    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerStarted","Data":"13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b"}
May 20 15:33:06.549214 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.549160    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k" event={"ID":"fc05d954-00b2-4086-a171-f90af0bdaf0f","Type":"ContainerStarted","Data":"68057c2f653ad3ea48a1439870c40f7c391b5f6163478d786f63f73cac30b3ea"}
May 20 15:33:06.550842 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.550793    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-xvk6q" podStartSLOduration=1.619143673 podStartE2EDuration="20.550777002s" podCreationTimestamp="2026-05-20 15:32:46 +0000 UTC" firstStartedPulling="2026-05-20 15:32:47.401806223 +0000 UTC m=+57.105899791" lastFinishedPulling="2026-05-20 15:33:06.333439537 +0000 UTC m=+76.037533120" observedRunningTime="2026-05-20 15:33:06.544977807 +0000 UTC m=+76.249071399" watchObservedRunningTime="2026-05-20 15:33:06.550777002 +0000 UTC m=+76.254870593"
May 20 15:33:06.566530 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.566469    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-568c5f6977-frpl4"]
May 20 15:33:06.566841 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:06.566802    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp" podStartSLOduration=0.997363805 podStartE2EDuration="11.566789804s" podCreationTimestamp="2026-05-20 15:32:55 +0000 UTC" firstStartedPulling="2026-05-20 15:32:55.65881192 +0000 UTC m=+65.362905491" lastFinishedPulling="2026-05-20 15:33:06.228237913 +0000 UTC m=+75.932331490" observedRunningTime="2026-05-20 15:33:06.565393841 +0000 UTC m=+76.269487433" watchObservedRunningTime="2026-05-20 15:33:06.566789804 +0000 UTC m=+76.270883393"
May 20 15:33:06.570916 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:33:06.570887    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80cf51f_e753_4278_8d78_65728dac7bdc.slice/crio-f33e0f7286c67859802bbcc2d1b2401784b0e8735d8836e2b2d5acf62ac2b02d WatchSource:0}: Error finding container f33e0f7286c67859802bbcc2d1b2401784b0e8735d8836e2b2d5acf62ac2b02d: Status 404 returned error can't find the container with id f33e0f7286c67859802bbcc2d1b2401784b0e8735d8836e2b2d5acf62ac2b02d
May 20 15:33:07.557153 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.556982    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q4gpd" event={"ID":"ddb4cb56-7904-4742-ac26-ecbd4f38df78","Type":"ContainerStarted","Data":"3d7a7c6f2ad88f3d052d15dd83ecd11ca1b17fd1be5e91bd574bb5d7fd72d7ee"}
May 20 15:33:07.564206 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.564111    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp" event={"ID":"a46d6274-fc02-4bea-8ba5-31040d263711","Type":"ContainerStarted","Data":"a5496f533608daba29a15477e76cfcb3c63d1bd75a85361fd40ef03646eae4b7"}
May 20 15:33:07.570345 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.568854    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-46dsm" event={"ID":"9f8d9d11-6bdb-402e-ab74-7485d6c52a79","Type":"ContainerStarted","Data":"6898f13da523cfd51dd81d6958d7add406355281076fa55dc09108a4c49874de"}
May 20 15:33:07.580545 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.580425    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerStarted","Data":"4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c"}
May 20 15:33:07.580545 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.580470    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerStarted","Data":"d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5"}
May 20 15:33:07.580545 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.580484    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerStarted","Data":"1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982"}
May 20 15:33:07.580545 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.580496    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerStarted","Data":"b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6"}
May 20 15:33:07.586926 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.586346    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k" event={"ID":"fc05d954-00b2-4086-a171-f90af0bdaf0f","Type":"ContainerStarted","Data":"e5749a18dda490c2d0d63e60f3639bf4128101481993f17366c7c930691a4ff9"}
May 20 15:33:07.586926 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.586394    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k" event={"ID":"fc05d954-00b2-4086-a171-f90af0bdaf0f","Type":"ContainerStarted","Data":"4e630cd71bbc2478de906d9478e0a9ca947a4dc5a7ecf7c869f8cbe885a86345"}
May 20 15:33:07.602905 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.601823    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-568c5f6977-frpl4" event={"ID":"f80cf51f-e753-4278-8d78-65728dac7bdc","Type":"ContainerStarted","Data":"d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f"}
May 20 15:33:07.602905 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.601881    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-568c5f6977-frpl4" event={"ID":"f80cf51f-e753-4278-8d78-65728dac7bdc","Type":"ContainerStarted","Data":"f33e0f7286c67859802bbcc2d1b2401784b0e8735d8836e2b2d5acf62ac2b02d"}
May 20 15:33:07.609772 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.609731    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-xvk6q"
May 20 15:33:07.641509 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:07.640390    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-568c5f6977-frpl4" podStartSLOduration=6.64036759 podStartE2EDuration="6.64036759s" podCreationTimestamp="2026-05-20 15:33:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:33:07.620132666 +0000 UTC m=+77.324226257" watchObservedRunningTime="2026-05-20 15:33:07.64036759 +0000 UTC m=+77.344461183"
May 20 15:33:12.297450 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:12.297407    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:12.297941 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:12.297469    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:12.302965 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:12.302940    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:12.630910 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:12.630787    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k" event={"ID":"fc05d954-00b2-4086-a171-f90af0bdaf0f","Type":"ContainerStarted","Data":"62a9f97d88ea426a73dcfa0e5dda1cda8f9140f74c7eaa47da6eeb6b09bc6e3f"}
May 20 15:33:12.639449 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:12.639141    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:12.706542 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:12.706457    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65cbc4f8b8-6jzs8"]
May 20 15:33:13.643776 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.643739    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g" event={"ID":"1fe0f182-2f5b-41bc-a208-6df929d947da","Type":"ContainerStarted","Data":"c2afce464f72bea1102a430c928940f7bd23a23c744ce5ce336590ffa3d786cd"}
May 20 15:33:13.643776 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.643784    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g" event={"ID":"1fe0f182-2f5b-41bc-a208-6df929d947da","Type":"ContainerStarted","Data":"d6c546495153e5eb82d6fcab6c89febc61ac652143ba9adb9a2b1da75c85de8e"}
May 20 15:33:13.644451 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.643799    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g" event={"ID":"1fe0f182-2f5b-41bc-a208-6df929d947da","Type":"ContainerStarted","Data":"6325dba78fb9733875510837dc58d0f0a75c5a62fc732c731c2b6e9ed5eedbaa"}
May 20 15:33:13.645548 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.645511    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-wvntl" event={"ID":"3a204308-a018-4037-a3db-8bc7a0e5444c","Type":"ContainerStarted","Data":"7892af3a3fe3cbfcb38be4c422f7be16f079ca429de1ba75b8e01a53f3af8b68"}
May 20 15:33:13.645865 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.645835    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-wvntl"
May 20 15:33:13.649241 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.649213    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerStarted","Data":"cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949"}
May 20 15:33:13.651792 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.651756    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-wvntl"
May 20 15:33:13.653092 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.653064    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k" event={"ID":"fc05d954-00b2-4086-a171-f90af0bdaf0f","Type":"ContainerStarted","Data":"8fa7324933b756a671ae3eaa43147a18bb1b94d04f428c22f3bc1234c3df20f8"}
May 20 15:33:13.653283 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.653100    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k" event={"ID":"fc05d954-00b2-4086-a171-f90af0bdaf0f","Type":"ContainerStarted","Data":"5fd75e129ba3df7bb73b3f56277ed8fdb0768c1b617257d8224ddbf982eb31e4"}
May 20 15:33:13.653351 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.653302    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:33:13.654787 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.654732    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q4gpd" event={"ID":"ddb4cb56-7904-4742-ac26-ecbd4f38df78","Type":"ContainerStarted","Data":"bdcc7136b49b847220e9ee2dde71077e5bafd23f01b435ecb77bae29510f03ba"}
May 20 15:33:13.654898 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.654881    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:33:13.656708 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.656684    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-46dsm" event={"ID":"9f8d9d11-6bdb-402e-ab74-7485d6c52a79","Type":"ContainerStarted","Data":"ac5ea24f4e2081168afd150b4c6a3c3b881a491964d75a8fdeb40a8861a17a53"}
May 20 15:33:13.656818 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.656715    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-46dsm" event={"ID":"9f8d9d11-6bdb-402e-ab74-7485d6c52a79","Type":"ContainerStarted","Data":"f8ee52d0a4c233a8535c587196194418ae7cca3100f5612d3735615eda1e6d00"}
May 20 15:33:13.660198 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.660177    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k"
May 20 15:33:13.671069 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.671009    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6bc566fb97-8bt9g" podStartSLOduration=11.708876326 podStartE2EDuration="17.67099202s" podCreationTimestamp="2026-05-20 15:32:56 +0000 UTC" firstStartedPulling="2026-05-20 15:33:06.48104863 +0000 UTC m=+76.185142205" lastFinishedPulling="2026-05-20 15:33:12.443164326 +0000 UTC m=+82.147257899" observedRunningTime="2026-05-20 15:33:13.668019467 +0000 UTC m=+83.372113059" watchObservedRunningTime="2026-05-20 15:33:13.67099202 +0000 UTC m=+83.375085612"
May 20 15:33:13.694147 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.693535    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-46dsm" podStartSLOduration=76.80271616 podStartE2EDuration="1m22.69351605s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:33:06.519321496 +0000 UTC m=+76.223415084" lastFinishedPulling="2026-05-20 15:33:12.410121401 +0000 UTC m=+82.114214974" observedRunningTime="2026-05-20 15:33:13.686483381 +0000 UTC m=+83.390576973" watchObservedRunningTime="2026-05-20 15:33:13.69351605 +0000 UTC m=+83.397609643"
May 20 15:33:13.711902 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.711849    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d796ddbc9-bbf6k" podStartSLOduration=3.029862633 podStartE2EDuration="20.711833923s" podCreationTimestamp="2026-05-20 15:32:53 +0000 UTC" firstStartedPulling="2026-05-20 15:32:54.728138377 +0000 UTC m=+64.432231946" lastFinishedPulling="2026-05-20 15:33:12.41010966 +0000 UTC m=+82.114203236" observedRunningTime="2026-05-20 15:33:13.710990436 +0000 UTC m=+83.415084064" watchObservedRunningTime="2026-05-20 15:33:13.711833923 +0000 UTC m=+83.415927506"
May 20 15:33:13.728523 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.728463    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-wvntl" podStartSLOduration=12.758262833 podStartE2EDuration="18.728448163s" podCreationTimestamp="2026-05-20 15:32:55 +0000 UTC" firstStartedPulling="2026-05-20 15:33:06.439937534 +0000 UTC m=+76.144031106" lastFinishedPulling="2026-05-20 15:33:12.410122868 +0000 UTC m=+82.114216436" observedRunningTime="2026-05-20 15:33:13.726546754 +0000 UTC m=+83.430640343" watchObservedRunningTime="2026-05-20 15:33:13.728448163 +0000 UTC m=+83.432541752"
May 20 15:33:13.784358 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.784296    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.758873613 podStartE2EDuration="22.784279401s" podCreationTimestamp="2026-05-20 15:32:51 +0000 UTC" firstStartedPulling="2026-05-20 15:32:53.384734046 +0000 UTC m=+63.088827617" lastFinishedPulling="2026-05-20 15:33:12.410139824 +0000 UTC m=+82.114233405" observedRunningTime="2026-05-20 15:33:13.759879622 +0000 UTC m=+83.463973213" watchObservedRunningTime="2026-05-20 15:33:13.784279401 +0000 UTC m=+83.488373014"
May 20 15:33:13.785495 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:13.785440    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q4gpd" podStartSLOduration=76.874645962 podStartE2EDuration="1m22.78542992s" podCreationTimestamp="2026-05-20 15:31:51 +0000 UTC" firstStartedPulling="2026-05-20 15:33:06.54443768 +0000 UTC m=+76.248531263" lastFinishedPulling="2026-05-20 15:33:12.455221649 +0000 UTC m=+82.159315221" observedRunningTime="2026-05-20 15:33:13.778613319 +0000 UTC m=+83.482706907" watchObservedRunningTime="2026-05-20 15:33:13.78542992 +0000 UTC m=+83.489523511"
May 20 15:33:14.717175 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.717135    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-fd5669758-fs6xv"]
May 20 15:33:14.773354 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.773316    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd5669758-fs6xv"]
May 20 15:33:14.773512 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.773485    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.845907 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.845863    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-oauth-config\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.846101 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.845923    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-oauth-serving-cert\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.846101 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.846015    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-service-ca\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.846101 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.846062    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-trusted-ca-bundle\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.846379 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.846164    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-console-config\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.846379 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.846220    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nrkv\" (UniqueName: \"kubernetes.io/projected/fc38d598-5188-44a6-8276-23a066d6cb5b-kube-api-access-2nrkv\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.846379 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.846260    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-serving-cert\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.947104 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.947056    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-trusted-ca-bundle\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.947304 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.947118    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-console-config\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.947304 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.947147    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2nrkv\" (UniqueName: \"kubernetes.io/projected/fc38d598-5188-44a6-8276-23a066d6cb5b-kube-api-access-2nrkv\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.947304 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.947182    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-serving-cert\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.947304 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.947246    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-oauth-config\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.947304 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.947286    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-oauth-serving-cert\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.947541 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.947328    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-service-ca\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.948070 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.948041    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-console-config\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.948180 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.948159    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-trusted-ca-bundle\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.948643 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.948617    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-service-ca\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.948859 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.948828    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-oauth-serving-cert\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.950480 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.950456    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-serving-cert\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.952048 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.952022    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-oauth-config\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:14.956973 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:14.956952    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nrkv\" (UniqueName: \"kubernetes.io/projected/fc38d598-5188-44a6-8276-23a066d6cb5b-kube-api-access-2nrkv\") pod \"console-fd5669758-fs6xv\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") " pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:15.085481 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:15.085372    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:15.113469 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:15.113423    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:33:15.244819 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:15.244777    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fd5669758-fs6xv"]
May 20 15:33:15.248118 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:33:15.248082    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc38d598_5188_44a6_8276_23a066d6cb5b.slice/crio-001f183e93b2320e88ebadc059fddeb94206083aaa9a98265f51ff95d84ede7c WatchSource:0}: Error finding container 001f183e93b2320e88ebadc059fddeb94206083aaa9a98265f51ff95d84ede7c: Status 404 returned error can't find the container with id 001f183e93b2320e88ebadc059fddeb94206083aaa9a98265f51ff95d84ede7c
May 20 15:33:15.479193 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:15.479146    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:33:15.479349 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:15.479214    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:33:15.670410 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:15.670328    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd5669758-fs6xv" event={"ID":"fc38d598-5188-44a6-8276-23a066d6cb5b","Type":"ContainerStarted","Data":"3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc"}
May 20 15:33:15.670410 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:15.670367    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd5669758-fs6xv" event={"ID":"fc38d598-5188-44a6-8276-23a066d6cb5b","Type":"ContainerStarted","Data":"001f183e93b2320e88ebadc059fddeb94206083aaa9a98265f51ff95d84ede7c"}
May 20 15:33:15.689994 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:15.689933    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-fd5669758-fs6xv" podStartSLOduration=1.6899145500000001 podStartE2EDuration="1.68991455s" podCreationTimestamp="2026-05-20 15:33:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:33:15.688551035 +0000 UTC m=+85.392644625" watchObservedRunningTime="2026-05-20 15:33:15.68991455 +0000 UTC m=+85.394008142"
May 20 15:33:25.086350 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:25.086300    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:25.086350 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:25.086352    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:25.091186 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:25.091161    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:25.721843 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:25.721813    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:33:25.769097 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:25.769066    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-568c5f6977-frpl4"]
May 20 15:33:35.485724 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:35.485436    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:33:35.489548 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:35.489525    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6d685b6d6c-pn6kp"
May 20 15:33:37.743624 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:37.743533    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-65cbc4f8b8-6jzs8" podUID="7c8e2758-cd74-40bc-851f-f74c33c0d5a6" containerName="console" containerID="cri-o://315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa" gracePeriod=15
May 20 15:33:38.018209 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.018185    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65cbc4f8b8-6jzs8_7c8e2758-cd74-40bc-851f-f74c33c0d5a6/console/0.log"
May 20 15:33:38.018346 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.018266    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:33:38.044317 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.044286    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-oauth-config\") pod \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") "
May 20 15:33:38.044473 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.044352    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-oauth-serving-cert\") pod \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") "
May 20 15:33:38.044526 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.044466    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbpl5\" (UniqueName: \"kubernetes.io/projected/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-kube-api-access-kbpl5\") pod \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") "
May 20 15:33:38.044526 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.044507    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-config\") pod \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") "
May 20 15:33:38.044647 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.044547    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-serving-cert\") pod \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") "
May 20 15:33:38.044647 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.044615    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-service-ca\") pod \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\" (UID: \"7c8e2758-cd74-40bc-851f-f74c33c0d5a6\") "
May 20 15:33:38.044851 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.044815    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7c8e2758-cd74-40bc-851f-f74c33c0d5a6" (UID: "7c8e2758-cd74-40bc-851f-f74c33c0d5a6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:38.045034 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.044927    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-oauth-serving-cert\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:38.045284 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.045250    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-service-ca" (OuterVolumeSpecName: "service-ca") pod "7c8e2758-cd74-40bc-851f-f74c33c0d5a6" (UID: "7c8e2758-cd74-40bc-851f-f74c33c0d5a6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:38.045284 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.045257    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-config" (OuterVolumeSpecName: "console-config") pod "7c8e2758-cd74-40bc-851f-f74c33c0d5a6" (UID: "7c8e2758-cd74-40bc-851f-f74c33c0d5a6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:38.047114 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.047039    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-kube-api-access-kbpl5" (OuterVolumeSpecName: "kube-api-access-kbpl5") pod "7c8e2758-cd74-40bc-851f-f74c33c0d5a6" (UID: "7c8e2758-cd74-40bc-851f-f74c33c0d5a6"). InnerVolumeSpecName "kube-api-access-kbpl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:33:38.047368 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.047349    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7c8e2758-cd74-40bc-851f-f74c33c0d5a6" (UID: "7c8e2758-cd74-40bc-851f-f74c33c0d5a6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:33:38.047516 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.047501    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7c8e2758-cd74-40bc-851f-f74c33c0d5a6" (UID: "7c8e2758-cd74-40bc-851f-f74c33c0d5a6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:33:38.145363 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.145323    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-config\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:38.145363 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.145360    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-serving-cert\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:38.145569 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.145375    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-service-ca\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:38.145569 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.145387    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-console-oauth-config\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:38.145569 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.145399    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbpl5\" (UniqueName: \"kubernetes.io/projected/7c8e2758-cd74-40bc-851f-f74c33c0d5a6-kube-api-access-kbpl5\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:38.761113 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.761087    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65cbc4f8b8-6jzs8_7c8e2758-cd74-40bc-851f-f74c33c0d5a6/console/0.log"
May 20 15:33:38.761523 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.761126    2567 generic.go:358] "Generic (PLEG): container finished" podID="7c8e2758-cd74-40bc-851f-f74c33c0d5a6" containerID="315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa" exitCode=2
May 20 15:33:38.761523 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.761186    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cbc4f8b8-6jzs8" event={"ID":"7c8e2758-cd74-40bc-851f-f74c33c0d5a6","Type":"ContainerDied","Data":"315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa"}
May 20 15:33:38.761523 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.761192    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65cbc4f8b8-6jzs8"
May 20 15:33:38.761523 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.761209    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65cbc4f8b8-6jzs8" event={"ID":"7c8e2758-cd74-40bc-851f-f74c33c0d5a6","Type":"ContainerDied","Data":"ad75975a781cf103c25d13f11d1fb5ad1d9712f951a21f3a184914633425cd01"}
May 20 15:33:38.761523 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.761223    2567 scope.go:117] "RemoveContainer" containerID="315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa"
May 20 15:33:38.783737 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.783703    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65cbc4f8b8-6jzs8"]
May 20 15:33:38.784436 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.784420    2567 scope.go:117] "RemoveContainer" containerID="315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa"
May 20 15:33:38.784774 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:33:38.784748    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa\": container with ID starting with 315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa not found: ID does not exist" containerID="315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa"
May 20 15:33:38.784833 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.784781    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa"} err="failed to get container status \"315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa\": rpc error: code = NotFound desc = could not find container \"315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa\": container with ID starting with 315d717a1a8bc9ec915735684b1ba3d2b42ee403df1ba23c40286e3c3f42e0aa not found: ID does not exist"
May 20 15:33:38.787404 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:38.787372    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65cbc4f8b8-6jzs8"]
May 20 15:33:39.061468 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:39.061374    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c8e2758-cd74-40bc-851f-f74c33c0d5a6" path="/var/lib/kubelet/pods/7c8e2758-cd74-40bc-851f-f74c33c0d5a6/volumes"
May 20 15:33:44.666738 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:44.666704    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q4gpd"
May 20 15:33:50.790501 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:50.790431    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-568c5f6977-frpl4" podUID="f80cf51f-e753-4278-8d78-65728dac7bdc" containerName="console" containerID="cri-o://d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f" gracePeriod=15
May 20 15:33:51.098009 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.097986    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-568c5f6977-frpl4_f80cf51f-e753-4278-8d78-65728dac7bdc/console/0.log"
May 20 15:33:51.098140 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.098049    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:51.156378 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.156339    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-serving-cert\") pod \"f80cf51f-e753-4278-8d78-65728dac7bdc\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") "
May 20 15:33:51.156378 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.156379    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-trusted-ca-bundle\") pod \"f80cf51f-e753-4278-8d78-65728dac7bdc\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") "
May 20 15:33:51.156648 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.156448    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-service-ca\") pod \"f80cf51f-e753-4278-8d78-65728dac7bdc\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") "
May 20 15:33:51.156648 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.156476    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-oauth-config\") pod \"f80cf51f-e753-4278-8d78-65728dac7bdc\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") "
May 20 15:33:51.156648 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.156515    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-console-config\") pod \"f80cf51f-e753-4278-8d78-65728dac7bdc\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") "
May 20 15:33:51.156648 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.156545    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj5pn\" (UniqueName: \"kubernetes.io/projected/f80cf51f-e753-4278-8d78-65728dac7bdc-kube-api-access-sj5pn\") pod \"f80cf51f-e753-4278-8d78-65728dac7bdc\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") "
May 20 15:33:51.156648 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.156573    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-oauth-serving-cert\") pod \"f80cf51f-e753-4278-8d78-65728dac7bdc\" (UID: \"f80cf51f-e753-4278-8d78-65728dac7bdc\") "
May 20 15:33:51.156987 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.156953    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-service-ca" (OuterVolumeSpecName: "service-ca") pod "f80cf51f-e753-4278-8d78-65728dac7bdc" (UID: "f80cf51f-e753-4278-8d78-65728dac7bdc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:51.157341 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.156959    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-console-config" (OuterVolumeSpecName: "console-config") pod "f80cf51f-e753-4278-8d78-65728dac7bdc" (UID: "f80cf51f-e753-4278-8d78-65728dac7bdc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:51.157433 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.156983    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f80cf51f-e753-4278-8d78-65728dac7bdc" (UID: "f80cf51f-e753-4278-8d78-65728dac7bdc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:51.157433 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.157361    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f80cf51f-e753-4278-8d78-65728dac7bdc" (UID: "f80cf51f-e753-4278-8d78-65728dac7bdc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:33:51.158827 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.158794    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f80cf51f-e753-4278-8d78-65728dac7bdc" (UID: "f80cf51f-e753-4278-8d78-65728dac7bdc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:33:51.158908 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.158819    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f80cf51f-e753-4278-8d78-65728dac7bdc" (UID: "f80cf51f-e753-4278-8d78-65728dac7bdc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:33:51.158908 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.158829    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80cf51f-e753-4278-8d78-65728dac7bdc-kube-api-access-sj5pn" (OuterVolumeSpecName: "kube-api-access-sj5pn") pod "f80cf51f-e753-4278-8d78-65728dac7bdc" (UID: "f80cf51f-e753-4278-8d78-65728dac7bdc"). InnerVolumeSpecName "kube-api-access-sj5pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:33:51.257552 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.257519    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-serving-cert\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:51.257552 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.257548    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-trusted-ca-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:51.257552 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.257557    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-service-ca\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:51.257801 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.257567    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f80cf51f-e753-4278-8d78-65728dac7bdc-console-oauth-config\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:51.257801 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.257577    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-console-config\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:51.257801 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.257605    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj5pn\" (UniqueName: \"kubernetes.io/projected/f80cf51f-e753-4278-8d78-65728dac7bdc-kube-api-access-sj5pn\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:51.257801 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.257615    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f80cf51f-e753-4278-8d78-65728dac7bdc-oauth-serving-cert\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:33:51.804376 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.804348    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-568c5f6977-frpl4_f80cf51f-e753-4278-8d78-65728dac7bdc/console/0.log"
May 20 15:33:51.804822 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.804386    2567 generic.go:358] "Generic (PLEG): container finished" podID="f80cf51f-e753-4278-8d78-65728dac7bdc" containerID="d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f" exitCode=2
May 20 15:33:51.804822 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.804421    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-568c5f6977-frpl4" event={"ID":"f80cf51f-e753-4278-8d78-65728dac7bdc","Type":"ContainerDied","Data":"d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f"}
May 20 15:33:51.804822 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.804449    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-568c5f6977-frpl4"
May 20 15:33:51.804822 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.804466    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-568c5f6977-frpl4" event={"ID":"f80cf51f-e753-4278-8d78-65728dac7bdc","Type":"ContainerDied","Data":"f33e0f7286c67859802bbcc2d1b2401784b0e8735d8836e2b2d5acf62ac2b02d"}
May 20 15:33:51.804822 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.804490    2567 scope.go:117] "RemoveContainer" containerID="d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f"
May 20 15:33:51.813071 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.813050    2567 scope.go:117] "RemoveContainer" containerID="d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f"
May 20 15:33:51.813308 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:33:51.813290    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f\": container with ID starting with d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f not found: ID does not exist" containerID="d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f"
May 20 15:33:51.813357 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.813318    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f"} err="failed to get container status \"d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f\": rpc error: code = NotFound desc = could not find container \"d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f\": container with ID starting with d219c04fcd7f5360f522b88ab94e639c8f8476cac6653b5dc354b8eac1b2324f not found: ID does not exist"
May 20 15:33:51.825925 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.825901    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-568c5f6977-frpl4"]
May 20 15:33:51.829464 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:51.829439    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-568c5f6977-frpl4"]
May 20 15:33:53.056539 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:33:53.056507    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f80cf51f-e753-4278-8d78-65728dac7bdc" path="/var/lib/kubelet/pods/f80cf51f-e753-4278-8d78-65728dac7bdc/volumes"
May 20 15:34:11.255918 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.255867    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:34:11.256536 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.256489    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="alertmanager" containerID="cri-o://13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b" gracePeriod=120
May 20 15:34:11.256642 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.256539    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy-metric" containerID="cri-o://4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c" gracePeriod=120
May 20 15:34:11.256642 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.256567    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy-web" containerID="cri-o://1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982" gracePeriod=120
May 20 15:34:11.256745 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.256612    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy" containerID="cri-o://d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5" gracePeriod=120
May 20 15:34:11.256745 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.256655    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="prom-label-proxy" containerID="cri-o://cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949" gracePeriod=120
May 20 15:34:11.256745 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.256661    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="config-reloader" containerID="cri-o://b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6" gracePeriod=120
May 20 15:34:11.863357 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.863319    2567 generic.go:358] "Generic (PLEG): container finished" podID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerID="cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949" exitCode=0
May 20 15:34:11.863357 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.863348    2567 generic.go:358] "Generic (PLEG): container finished" podID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerID="4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c" exitCode=0
May 20 15:34:11.863357 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.863357    2567 generic.go:358] "Generic (PLEG): container finished" podID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerID="d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5" exitCode=0
May 20 15:34:11.863357 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.863365    2567 generic.go:358] "Generic (PLEG): container finished" podID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerID="b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6" exitCode=0
May 20 15:34:11.863653 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.863375    2567 generic.go:358] "Generic (PLEG): container finished" podID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerID="13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b" exitCode=0
May 20 15:34:11.863653 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.863386    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerDied","Data":"cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949"}
May 20 15:34:11.863653 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.863419    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerDied","Data":"4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c"}
May 20 15:34:11.863653 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.863429    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerDied","Data":"d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5"}
May 20 15:34:11.863653 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.863438    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerDied","Data":"b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6"}
May 20 15:34:11.863653 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:11.863446    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerDied","Data":"13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b"}
May 20 15:34:12.497562 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.497534    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:12.646495 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646453    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.646495 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646503    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-main-db\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.646769 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646521    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-metrics-client-ca\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.646769 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646566    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-web-config\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.646769 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646624    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-out\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.646769 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646666    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-trusted-ca-bundle\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.646769 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646706    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-main-tls\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.646769 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646737    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-cluster-tls-config\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.647073 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646844    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-volume\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.647073 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646882    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vndv6\" (UniqueName: \"kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-kube-api-access-vndv6\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.647073 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646892    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:34:12.647073 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646919    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-web\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.647073 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646953    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.647073 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646976    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:34:12.647073 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.646994    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-tls-assets\") pod \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\" (UID: \"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6\") "
May 20 15:34:12.647418 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.647239    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-main-db\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.647418 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.647259    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-metrics-client-ca\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.647418 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.647261    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:34:12.649718 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.649679    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:34:12.650228 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.649985    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-volume" (OuterVolumeSpecName: "config-volume") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:34:12.650228 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.650016    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:34:12.650373 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.650262    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:34:12.650373 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.650301    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-out" (OuterVolumeSpecName: "config-out") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:34:12.650373 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.650313    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:34:12.652159 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.652137    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:34:12.652541 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.652514    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-kube-api-access-vndv6" (OuterVolumeSpecName: "kube-api-access-vndv6") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "kube-api-access-vndv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:34:12.654942 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.654698    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:34:12.660782 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.660756    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-web-config" (OuterVolumeSpecName: "web-config") pod "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" (UID: "d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:34:12.748478 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.748432    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-main-tls\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.748478 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.748470    2567 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-cluster-tls-config\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.748478 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.748484    2567 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-volume\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.748761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.748522    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vndv6\" (UniqueName: \"kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-kube-api-access-vndv6\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.748761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.748536    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.748761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.748550    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.748761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.748562    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-tls-assets\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.748761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.748575    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.748761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.748613    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-web-config\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.748761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.748625    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-config-out\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.748761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.748638    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:34:12.868461 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.868428    2567 generic.go:358] "Generic (PLEG): container finished" podID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerID="1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982" exitCode=0
May 20 15:34:12.868656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.868476    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerDied","Data":"1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982"}
May 20 15:34:12.868656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.868504    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6","Type":"ContainerDied","Data":"861b9590b1ba909d5f8b228cbf635cd25d6aefd5614b8ef31fc33ebd76c95e5b"}
May 20 15:34:12.868656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.868520    2567 scope.go:117] "RemoveContainer" containerID="cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949"
May 20 15:34:12.868656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.868530    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:12.876853 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.876828    2567 scope.go:117] "RemoveContainer" containerID="4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c"
May 20 15:34:12.884164 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.884145    2567 scope.go:117] "RemoveContainer" containerID="d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5"
May 20 15:34:12.891079 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.891057    2567 scope.go:117] "RemoveContainer" containerID="1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982"
May 20 15:34:12.894340 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.894309    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:34:12.899570 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.899530    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:34:12.899997 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.899974    2567 scope.go:117] "RemoveContainer" containerID="b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6"
May 20 15:34:12.906927 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.906904    2567 scope.go:117] "RemoveContainer" containerID="13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b"
May 20 15:34:12.913865 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.913845    2567 scope.go:117] "RemoveContainer" containerID="e38d9a9976271eb37738d0818da5f2ce0acbdd211f19d079eabb4318cf450a10"
May 20 15:34:12.920769 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.920682    2567 scope.go:117] "RemoveContainer" containerID="cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949"
May 20 15:34:12.920953 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:34:12.920933    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949\": container with ID starting with cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949 not found: ID does not exist" containerID="cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949"
May 20 15:34:12.920993 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.920962    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949"} err="failed to get container status \"cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949\": rpc error: code = NotFound desc = could not find container \"cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949\": container with ID starting with cad3bbb1e14fa8b4a16d43655a99d5aea0d070835a899cb0177ebdadcb08d949 not found: ID does not exist"
May 20 15:34:12.920993 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.920983    2567 scope.go:117] "RemoveContainer" containerID="4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c"
May 20 15:34:12.921220 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:34:12.921203    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c\": container with ID starting with 4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c not found: ID does not exist" containerID="4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c"
May 20 15:34:12.921259 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.921228    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c"} err="failed to get container status \"4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c\": rpc error: code = NotFound desc = could not find container \"4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c\": container with ID starting with 4210e5844fd43e235b4be71dabad4afbc028c4ea5679e531929533882b008f3c not found: ID does not exist"
May 20 15:34:12.921259 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.921245    2567 scope.go:117] "RemoveContainer" containerID="d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5"
May 20 15:34:12.921482 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:34:12.921466    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5\": container with ID starting with d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5 not found: ID does not exist" containerID="d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5"
May 20 15:34:12.921533 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.921485    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5"} err="failed to get container status \"d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5\": rpc error: code = NotFound desc = could not find container \"d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5\": container with ID starting with d07b589d757ae0b9ad85288d4eb0bf1d137884f52a7a0b651edaf8f554e4e5b5 not found: ID does not exist"
May 20 15:34:12.921533 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.921500    2567 scope.go:117] "RemoveContainer" containerID="1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982"
May 20 15:34:12.921751 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:34:12.921735    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982\": container with ID starting with 1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982 not found: ID does not exist" containerID="1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982"
May 20 15:34:12.926037 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.922057    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982"} err="failed to get container status \"1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982\": rpc error: code = NotFound desc = could not find container \"1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982\": container with ID starting with 1d7191944b1e8543df4fc709971e821dee243f4dff35b0b2d4b8e85fa3c8f982 not found: ID does not exist"
May 20 15:34:12.926037 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.922089    2567 scope.go:117] "RemoveContainer" containerID="b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6"
May 20 15:34:12.927657 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:34:12.926865    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6\": container with ID starting with b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6 not found: ID does not exist" containerID="b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6"
May 20 15:34:12.927657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.926923    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6"} err="failed to get container status \"b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6\": rpc error: code = NotFound desc = could not find container \"b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6\": container with ID starting with b8cc50dd5316600b75bf4a3461004b6b22157ce9297b71083fa0d694b7b9add6 not found: ID does not exist"
May 20 15:34:12.927657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.926949    2567 scope.go:117] "RemoveContainer" containerID="13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b"
May 20 15:34:12.927929 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:34:12.927797    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b\": container with ID starting with 13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b not found: ID does not exist" containerID="13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b"
May 20 15:34:12.927929 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.927830    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b"} err="failed to get container status \"13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b\": rpc error: code = NotFound desc = could not find container \"13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b\": container with ID starting with 13afb2a9f2224e09ac2242a68b0a4c0c9e1066214e272df6c71e2cb1d533d44b not found: ID does not exist"
May 20 15:34:12.928338 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.928090    2567 scope.go:117] "RemoveContainer" containerID="e38d9a9976271eb37738d0818da5f2ce0acbdd211f19d079eabb4318cf450a10"
May 20 15:34:12.928668 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:34:12.928648    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38d9a9976271eb37738d0818da5f2ce0acbdd211f19d079eabb4318cf450a10\": container with ID starting with e38d9a9976271eb37738d0818da5f2ce0acbdd211f19d079eabb4318cf450a10 not found: ID does not exist" containerID="e38d9a9976271eb37738d0818da5f2ce0acbdd211f19d079eabb4318cf450a10"
May 20 15:34:12.928747 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.928677    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38d9a9976271eb37738d0818da5f2ce0acbdd211f19d079eabb4318cf450a10"} err="failed to get container status \"e38d9a9976271eb37738d0818da5f2ce0acbdd211f19d079eabb4318cf450a10\": rpc error: code = NotFound desc = could not find container \"e38d9a9976271eb37738d0818da5f2ce0acbdd211f19d079eabb4318cf450a10\": container with ID starting with e38d9a9976271eb37738d0818da5f2ce0acbdd211f19d079eabb4318cf450a10 not found: ID does not exist"
May 20 15:34:12.929781 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.929753    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:34:12.930255 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930240    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="init-config-reloader"
May 20 15:34:12.930295 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930261    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="init-config-reloader"
May 20 15:34:12.930295 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930274    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="prom-label-proxy"
May 20 15:34:12.930295 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930280    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="prom-label-proxy"
May 20 15:34:12.930295 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930291    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="config-reloader"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930297    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="config-reloader"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930303    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930308    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930314    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8e2758-cd74-40bc-851f-f74c33c0d5a6" containerName="console"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930319    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8e2758-cd74-40bc-851f-f74c33c0d5a6" containerName="console"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930325    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy-metric"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930332    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy-metric"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930350    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy-web"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930355    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy-web"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930361    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="alertmanager"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930365    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="alertmanager"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930371    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f80cf51f-e753-4278-8d78-65728dac7bdc" containerName="console"
May 20 15:34:12.930413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930377    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f80cf51f-e753-4278-8d78-65728dac7bdc" containerName="console"
May 20 15:34:12.930775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930425    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy-web"
May 20 15:34:12.930775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930435    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="config-reloader"
May 20 15:34:12.930775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930441    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="f80cf51f-e753-4278-8d78-65728dac7bdc" containerName="console"
May 20 15:34:12.930775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930448    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="prom-label-proxy"
May 20 15:34:12.930775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930454    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy"
May 20 15:34:12.930775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930460    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="alertmanager"
May 20 15:34:12.930775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930466    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" containerName="kube-rbac-proxy-metric"
May 20 15:34:12.930775 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.930472    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="7c8e2758-cd74-40bc-851f-f74c33c0d5a6" containerName="console"
May 20 15:34:12.933245 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.933227    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:12.935961 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.935935    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 20 15:34:12.936164 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.936014    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 20 15:34:12.936267 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.936226    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gccw9\""
May 20 15:34:12.936267 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.936230    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 20 15:34:12.936401 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.936229    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 20 15:34:12.936401 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.936343    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 20 15:34:12.936401 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.936354    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 20 15:34:12.936549 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.936228    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 20 15:34:12.936635 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.936602    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 20 15:34:12.942363 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.941763    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 20 15:34:12.944496 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:12.944473    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:34:13.051726 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.051692    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.051726 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.051729    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25db283d-ab2f-4d00-b513-7a50b9481b91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.051953 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.051750    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25db283d-ab2f-4d00-b513-7a50b9481b91-config-out\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.051953 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.051840    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-config-volume\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.051953 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.051873    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25db283d-ab2f-4d00-b513-7a50b9481b91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.051953 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.051897    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25db283d-ab2f-4d00-b513-7a50b9481b91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.051953 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.051932    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25db283d-ab2f-4d00-b513-7a50b9481b91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.052118 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.051975    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-web-config\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.052118 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.051997    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.052118 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.052035    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.052118 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.052060    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.052118 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.052111    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.052283 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.052126    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q798\" (UniqueName: \"kubernetes.io/projected/25db283d-ab2f-4d00-b513-7a50b9481b91-kube-api-access-6q798\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.055926 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.055890    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6" path="/var/lib/kubelet/pods/d4f7ece4-fb90-4a62-8ff1-b1f0136d8be6/volumes"
May 20 15:34:13.152681 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.152578    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.152681 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.152648    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6q798\" (UniqueName: \"kubernetes.io/projected/25db283d-ab2f-4d00-b513-7a50b9481b91-kube-api-access-6q798\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.152857 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.152708    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.152910 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.152880    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25db283d-ab2f-4d00-b513-7a50b9481b91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.152963 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.152931    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25db283d-ab2f-4d00-b513-7a50b9481b91-config-out\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.153013 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.152967    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-config-volume\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.153013 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.152995    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25db283d-ab2f-4d00-b513-7a50b9481b91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.153109 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.153027    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25db283d-ab2f-4d00-b513-7a50b9481b91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.153109 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.153064    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25db283d-ab2f-4d00-b513-7a50b9481b91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.153109 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.153100    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-web-config\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.153251 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.153132    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.153251 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.153159    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.153251 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.153185    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.154276 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.154247    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25db283d-ab2f-4d00-b513-7a50b9481b91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.154413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.154362    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25db283d-ab2f-4d00-b513-7a50b9481b91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.154484 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.154436    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25db283d-ab2f-4d00-b513-7a50b9481b91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.157093 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.156664    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25db283d-ab2f-4d00-b513-7a50b9481b91-config-out\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.157093 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.157042    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.157324 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.157290    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-config-volume\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.157566 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.157542    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.157906 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.157873    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25db283d-ab2f-4d00-b513-7a50b9481b91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.157996 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.157949    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-web-config\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.158060 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.157998    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.158060 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.158041    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.158351 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.158330    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25db283d-ab2f-4d00-b513-7a50b9481b91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.161661 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.161638    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q798\" (UniqueName: \"kubernetes.io/projected/25db283d-ab2f-4d00-b513-7a50b9481b91-kube-api-access-6q798\") pod \"alertmanager-main-0\" (UID: \"25db283d-ab2f-4d00-b513-7a50b9481b91\") " pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.245134 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.245092    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 20 15:34:13.386711 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.386338    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 20 15:34:13.389191 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:34:13.389160    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25db283d_ab2f_4d00_b513_7a50b9481b91.slice/crio-62c4dcbe06aa40125b6d923dd42977bcb85113ca7c2dffde8692710f55251dff WatchSource:0}: Error finding container 62c4dcbe06aa40125b6d923dd42977bcb85113ca7c2dffde8692710f55251dff: Status 404 returned error can't find the container with id 62c4dcbe06aa40125b6d923dd42977bcb85113ca7c2dffde8692710f55251dff
May 20 15:34:13.876111 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.876072    2567 generic.go:358] "Generic (PLEG): container finished" podID="25db283d-ab2f-4d00-b513-7a50b9481b91" containerID="1c3cbeaca8854a40c460cfda33d160a31fdf02ef36ea61457ea546ca1877b1e3" exitCode=0
May 20 15:34:13.876499 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.876129    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25db283d-ab2f-4d00-b513-7a50b9481b91","Type":"ContainerDied","Data":"1c3cbeaca8854a40c460cfda33d160a31fdf02ef36ea61457ea546ca1877b1e3"}
May 20 15:34:13.876499 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:13.876152    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25db283d-ab2f-4d00-b513-7a50b9481b91","Type":"ContainerStarted","Data":"62c4dcbe06aa40125b6d923dd42977bcb85113ca7c2dffde8692710f55251dff"}
May 20 15:34:14.882543 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:14.882504    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25db283d-ab2f-4d00-b513-7a50b9481b91","Type":"ContainerStarted","Data":"567067e716e32264f2848b1c5a2fa9e939cc41a691d8758e097a9da55068c18c"}
May 20 15:34:14.882543 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:14.882546    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25db283d-ab2f-4d00-b513-7a50b9481b91","Type":"ContainerStarted","Data":"e702c1d19899e08a52f921969a9b29e692062187af1ed687d0397c5b2d72ccae"}
May 20 15:34:14.883040 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:14.882560    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25db283d-ab2f-4d00-b513-7a50b9481b91","Type":"ContainerStarted","Data":"99b3af6a3ea72fab5253dba7fcfcda86dfca2476794fbe9b9588a5a29508470f"}
May 20 15:34:14.883040 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:14.882572    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25db283d-ab2f-4d00-b513-7a50b9481b91","Type":"ContainerStarted","Data":"5b376d48e9ec96c707f528b737796aa63af3b9daa9f454a0645d5a71c3fa383d"}
May 20 15:34:14.883040 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:14.882615    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25db283d-ab2f-4d00-b513-7a50b9481b91","Type":"ContainerStarted","Data":"360e47cabda099e4a26a662cb6e5d47a83a310780fd57b15865caf621866ebe3"}
May 20 15:34:14.883040 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:14.882627    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25db283d-ab2f-4d00-b513-7a50b9481b91","Type":"ContainerStarted","Data":"5355a97e9492424c78f2b012ccd827b9e671109c86f2bae1c085661614a80eb4"}
May 20 15:34:14.913954 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:14.913869    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.913850145 podStartE2EDuration="2.913850145s" podCreationTimestamp="2026-05-20 15:34:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:34:14.911826291 +0000 UTC m=+144.615919907" watchObservedRunningTime="2026-05-20 15:34:14.913850145 +0000 UTC m=+144.617943738"
May 20 15:34:40.487898 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.487812    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-j6glz"]
May 20 15:34:40.490031 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.490006    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6glz"
May 20 15:34:40.492658 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.492639    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 20 15:34:40.499315 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.499286    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j6glz"]
May 20 15:34:40.576341 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.576298    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2c2975f-9436-4c8c-a470-7a2c0d401942-original-pull-secret\") pod \"global-pull-secret-syncer-j6glz\" (UID: \"c2c2975f-9436-4c8c-a470-7a2c0d401942\") " pod="kube-system/global-pull-secret-syncer-j6glz"
May 20 15:34:40.576341 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.576337    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2c2975f-9436-4c8c-a470-7a2c0d401942-dbus\") pod \"global-pull-secret-syncer-j6glz\" (UID: \"c2c2975f-9436-4c8c-a470-7a2c0d401942\") " pod="kube-system/global-pull-secret-syncer-j6glz"
May 20 15:34:40.576550 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.576429    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2c2975f-9436-4c8c-a470-7a2c0d401942-kubelet-config\") pod \"global-pull-secret-syncer-j6glz\" (UID: \"c2c2975f-9436-4c8c-a470-7a2c0d401942\") " pod="kube-system/global-pull-secret-syncer-j6glz"
May 20 15:34:40.591392 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.591358    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-59bfb67c-gzjbj"]
May 20 15:34:40.593616 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.593574    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.604181 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.604156    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59bfb67c-gzjbj"]
May 20 15:34:40.677544 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677507    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-trusted-ca-bundle\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.677742 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677561    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2c2975f-9436-4c8c-a470-7a2c0d401942-kubelet-config\") pod \"global-pull-secret-syncer-j6glz\" (UID: \"c2c2975f-9436-4c8c-a470-7a2c0d401942\") " pod="kube-system/global-pull-secret-syncer-j6glz"
May 20 15:34:40.677742 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677650    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2c2975f-9436-4c8c-a470-7a2c0d401942-kubelet-config\") pod \"global-pull-secret-syncer-j6glz\" (UID: \"c2c2975f-9436-4c8c-a470-7a2c0d401942\") " pod="kube-system/global-pull-secret-syncer-j6glz"
May 20 15:34:40.677742 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5l7\" (UniqueName: \"kubernetes.io/projected/604888dc-fa6b-4598-a157-919087886059-kube-api-access-2q5l7\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.677742 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677709    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-console-config\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.677742 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677741    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-service-ca\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.677911 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677758    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-serving-cert\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.677911 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677794    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2c2975f-9436-4c8c-a470-7a2c0d401942-original-pull-secret\") pod \"global-pull-secret-syncer-j6glz\" (UID: \"c2c2975f-9436-4c8c-a470-7a2c0d401942\") " pod="kube-system/global-pull-secret-syncer-j6glz"
May 20 15:34:40.677911 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677811    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-oauth-config\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.677911 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677829    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2c2975f-9436-4c8c-a470-7a2c0d401942-dbus\") pod \"global-pull-secret-syncer-j6glz\" (UID: \"c2c2975f-9436-4c8c-a470-7a2c0d401942\") " pod="kube-system/global-pull-secret-syncer-j6glz"
May 20 15:34:40.677911 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677850    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-oauth-serving-cert\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.678057 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.677999    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2c2975f-9436-4c8c-a470-7a2c0d401942-dbus\") pod \"global-pull-secret-syncer-j6glz\" (UID: \"c2c2975f-9436-4c8c-a470-7a2c0d401942\") " pod="kube-system/global-pull-secret-syncer-j6glz"
May 20 15:34:40.680250 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.680232    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2c2975f-9436-4c8c-a470-7a2c0d401942-original-pull-secret\") pod \"global-pull-secret-syncer-j6glz\" (UID: \"c2c2975f-9436-4c8c-a470-7a2c0d401942\") " pod="kube-system/global-pull-secret-syncer-j6glz"
May 20 15:34:40.778428 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.778325    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-console-config\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.778428 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.778375    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-service-ca\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.778428 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.778397    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-serving-cert\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.778746 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.778522    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-oauth-config\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.778746 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.778564    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-oauth-serving-cert\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.778746 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.778645    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-trusted-ca-bundle\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.778746 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.778719    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5l7\" (UniqueName: \"kubernetes.io/projected/604888dc-fa6b-4598-a157-919087886059-kube-api-access-2q5l7\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.779338 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.779149    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-console-config\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.779338 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.779200    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-service-ca\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.779338 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.779327    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-oauth-serving-cert\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.779530 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.779464    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-trusted-ca-bundle\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.781064 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.781045    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-serving-cert\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.781208 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.781187    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-oauth-config\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.788170 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.788144    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5l7\" (UniqueName: \"kubernetes.io/projected/604888dc-fa6b-4598-a157-919087886059-kube-api-access-2q5l7\") pod \"console-59bfb67c-gzjbj\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") " pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.798979 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.798957    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j6glz"
May 20 15:34:40.903219 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.903182    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:40.932377 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.932249    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j6glz"]
May 20 15:34:40.935013 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:34:40.934977    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c2975f_9436_4c8c_a470_7a2c0d401942.slice/crio-2446904b1cb94b2102c8071deccf87b52b35dbaf923e941a611c382d181a9e28 WatchSource:0}: Error finding container 2446904b1cb94b2102c8071deccf87b52b35dbaf923e941a611c382d181a9e28: Status 404 returned error can't find the container with id 2446904b1cb94b2102c8071deccf87b52b35dbaf923e941a611c382d181a9e28
May 20 15:34:40.964119 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:40.964072    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j6glz" event={"ID":"c2c2975f-9436-4c8c-a470-7a2c0d401942","Type":"ContainerStarted","Data":"2446904b1cb94b2102c8071deccf87b52b35dbaf923e941a611c382d181a9e28"}
May 20 15:34:41.039162 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:41.039134    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59bfb67c-gzjbj"]
May 20 15:34:41.041963 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:34:41.041933    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604888dc_fa6b_4598_a157_919087886059.slice/crio-de163dc8af249f8f053cec6d1c073d0f78568343fcb6689f1926354425b48598 WatchSource:0}: Error finding container de163dc8af249f8f053cec6d1c073d0f78568343fcb6689f1926354425b48598: Status 404 returned error can't find the container with id de163dc8af249f8f053cec6d1c073d0f78568343fcb6689f1926354425b48598
May 20 15:34:41.968763 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:41.968724    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bfb67c-gzjbj" event={"ID":"604888dc-fa6b-4598-a157-919087886059","Type":"ContainerStarted","Data":"115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370"}
May 20 15:34:41.968763 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:41.968769    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bfb67c-gzjbj" event={"ID":"604888dc-fa6b-4598-a157-919087886059","Type":"ContainerStarted","Data":"de163dc8af249f8f053cec6d1c073d0f78568343fcb6689f1926354425b48598"}
May 20 15:34:41.989705 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:41.989654    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-59bfb67c-gzjbj" podStartSLOduration=1.989636431 podStartE2EDuration="1.989636431s" podCreationTimestamp="2026-05-20 15:34:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:34:41.986297639 +0000 UTC m=+171.690391228" watchObservedRunningTime="2026-05-20 15:34:41.989636431 +0000 UTC m=+171.693730017"
May 20 15:34:44.979566 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:44.979471    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j6glz" event={"ID":"c2c2975f-9436-4c8c-a470-7a2c0d401942","Type":"ContainerStarted","Data":"fa298843977a5448c6403e29ac77f30b324250135c4f27ef60cac6ff16cb3910"}
May 20 15:34:44.997149 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:44.997100    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j6glz" podStartSLOduration=1.2875067470000001 podStartE2EDuration="4.997087146s" podCreationTimestamp="2026-05-20 15:34:40 +0000 UTC" firstStartedPulling="2026-05-20 15:34:40.93656061 +0000 UTC m=+170.640654178" lastFinishedPulling="2026-05-20 15:34:44.646141007 +0000 UTC m=+174.350234577" observedRunningTime="2026-05-20 15:34:44.995547296 +0000 UTC m=+174.699640900" watchObservedRunningTime="2026-05-20 15:34:44.997087146 +0000 UTC m=+174.701180736"
May 20 15:34:50.904322 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:50.904279    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:50.904322 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:50.904322    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:50.909897 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:50.909874    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:51.000340 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:51.000312    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:34:51.072567 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:34:51.072535    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd5669758-fs6xv"]
May 20 15:35:16.092493 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.092434    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-fd5669758-fs6xv" podUID="fc38d598-5188-44a6-8276-23a066d6cb5b" containerName="console" containerID="cri-o://3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc" gracePeriod=15
May 20 15:35:16.325188 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.325164    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-fd5669758-fs6xv_fc38d598-5188-44a6-8276-23a066d6cb5b/console/0.log"
May 20 15:35:16.325339 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.325239    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:35:16.387715 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.387611    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-oauth-serving-cert\") pod \"fc38d598-5188-44a6-8276-23a066d6cb5b\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") "
May 20 15:35:16.387715 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.387662    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-serving-cert\") pod \"fc38d598-5188-44a6-8276-23a066d6cb5b\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") "
May 20 15:35:16.387715 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.387711    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-trusted-ca-bundle\") pod \"fc38d598-5188-44a6-8276-23a066d6cb5b\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") "
May 20 15:35:16.387988 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.387736    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nrkv\" (UniqueName: \"kubernetes.io/projected/fc38d598-5188-44a6-8276-23a066d6cb5b-kube-api-access-2nrkv\") pod \"fc38d598-5188-44a6-8276-23a066d6cb5b\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") "
May 20 15:35:16.387988 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.387769    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-console-config\") pod \"fc38d598-5188-44a6-8276-23a066d6cb5b\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") "
May 20 15:35:16.387988 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.387801    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-service-ca\") pod \"fc38d598-5188-44a6-8276-23a066d6cb5b\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") "
May 20 15:35:16.387988 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.387834    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-oauth-config\") pod \"fc38d598-5188-44a6-8276-23a066d6cb5b\" (UID: \"fc38d598-5188-44a6-8276-23a066d6cb5b\") "
May 20 15:35:16.388168 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.388071    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fc38d598-5188-44a6-8276-23a066d6cb5b" (UID: "fc38d598-5188-44a6-8276-23a066d6cb5b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:35:16.388230 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.388201    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fc38d598-5188-44a6-8276-23a066d6cb5b" (UID: "fc38d598-5188-44a6-8276-23a066d6cb5b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:35:16.388361 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.388327    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-console-config" (OuterVolumeSpecName: "console-config") pod "fc38d598-5188-44a6-8276-23a066d6cb5b" (UID: "fc38d598-5188-44a6-8276-23a066d6cb5b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:35:16.388430 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.388418    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-service-ca" (OuterVolumeSpecName: "service-ca") pod "fc38d598-5188-44a6-8276-23a066d6cb5b" (UID: "fc38d598-5188-44a6-8276-23a066d6cb5b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:35:16.390141 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.390108    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fc38d598-5188-44a6-8276-23a066d6cb5b" (UID: "fc38d598-5188-44a6-8276-23a066d6cb5b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:35:16.390260 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.390185    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fc38d598-5188-44a6-8276-23a066d6cb5b" (UID: "fc38d598-5188-44a6-8276-23a066d6cb5b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:35:16.390260 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.390210    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc38d598-5188-44a6-8276-23a066d6cb5b-kube-api-access-2nrkv" (OuterVolumeSpecName: "kube-api-access-2nrkv") pod "fc38d598-5188-44a6-8276-23a066d6cb5b" (UID: "fc38d598-5188-44a6-8276-23a066d6cb5b"). InnerVolumeSpecName "kube-api-access-2nrkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:35:16.489447 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.489410    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-oauth-serving-cert\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:35:16.489447 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.489439    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-serving-cert\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:35:16.489447 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.489450    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-trusted-ca-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:35:16.489700 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.489459    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nrkv\" (UniqueName: \"kubernetes.io/projected/fc38d598-5188-44a6-8276-23a066d6cb5b-kube-api-access-2nrkv\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:35:16.489700 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.489470    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-console-config\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:35:16.489700 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.489480    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc38d598-5188-44a6-8276-23a066d6cb5b-service-ca\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:35:16.489700 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:16.489488    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc38d598-5188-44a6-8276-23a066d6cb5b-console-oauth-config\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:35:17.072411 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:17.072385    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-fd5669758-fs6xv_fc38d598-5188-44a6-8276-23a066d6cb5b/console/0.log"
May 20 15:35:17.072540 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:17.072423    2567 generic.go:358] "Generic (PLEG): container finished" podID="fc38d598-5188-44a6-8276-23a066d6cb5b" containerID="3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc" exitCode=2
May 20 15:35:17.072540 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:17.072453    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd5669758-fs6xv" event={"ID":"fc38d598-5188-44a6-8276-23a066d6cb5b","Type":"ContainerDied","Data":"3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc"}
May 20 15:35:17.072540 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:17.072492    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fd5669758-fs6xv" event={"ID":"fc38d598-5188-44a6-8276-23a066d6cb5b","Type":"ContainerDied","Data":"001f183e93b2320e88ebadc059fddeb94206083aaa9a98265f51ff95d84ede7c"}
May 20 15:35:17.072540 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:17.072498    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fd5669758-fs6xv"
May 20 15:35:17.072693 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:17.072503    2567 scope.go:117] "RemoveContainer" containerID="3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc"
May 20 15:35:17.080446 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:17.080426    2567 scope.go:117] "RemoveContainer" containerID="3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc"
May 20 15:35:17.080797 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:35:17.080777    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc\": container with ID starting with 3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc not found: ID does not exist" containerID="3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc"
May 20 15:35:17.080875 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:17.080807    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc"} err="failed to get container status \"3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc\": rpc error: code = NotFound desc = could not find container \"3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc\": container with ID starting with 3f6669ff30f06c66266587f09d254326d7d8ffb8268bab7efe30cc9ae2c3a4fc not found: ID does not exist"
May 20 15:35:17.090757 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:17.090729    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fd5669758-fs6xv"]
May 20 15:35:17.093309 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:17.093286    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fd5669758-fs6xv"]
May 20 15:35:19.058169 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:35:19.058122    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc38d598-5188-44a6-8276-23a066d6cb5b" path="/var/lib/kubelet/pods/fc38d598-5188-44a6-8276-23a066d6cb5b/volumes"
May 20 15:36:50.968834 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:36:50.968803    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:36:50.969309 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:36:50.968838    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:36:50.971477 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:36:50.971459    2567 kubelet.go:1628] "Image garbage collection succeeded"
May 20 15:38:20.508842 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.508804    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-9695b8856-x4klq"]
May 20 15:38:20.509311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.509146    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc38d598-5188-44a6-8276-23a066d6cb5b" containerName="console"
May 20 15:38:20.509311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.509157    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc38d598-5188-44a6-8276-23a066d6cb5b" containerName="console"
May 20 15:38:20.509311 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.509210    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="fc38d598-5188-44a6-8276-23a066d6cb5b" containerName="console"
May 20 15:38:20.512065 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.512049    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.532060 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.532023    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9695b8856-x4klq"]
May 20 15:38:20.602051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.602007    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-trusted-ca-bundle\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.602218 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.602060    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-oauth-serving-cert\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.602218 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.602155    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-console-serving-cert\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.602218 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.602195    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-service-ca\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.602328 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.602224    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbdtc\" (UniqueName: \"kubernetes.io/projected/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-kube-api-access-vbdtc\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.602328 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.602245    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-console-oauth-config\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.602328 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.602263    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-console-config\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.703678 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.703630    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-console-serving-cert\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.703678 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.703680    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-service-ca\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.703944 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.703707    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vbdtc\" (UniqueName: \"kubernetes.io/projected/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-kube-api-access-vbdtc\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.703944 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.703887    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-console-oauth-config\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.703944 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.703923    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-console-config\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.704109 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.703960    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-trusted-ca-bundle\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.704109 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.704029    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-oauth-serving-cert\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.704745 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.704634    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-console-config\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.704872 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.704764    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-service-ca\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.704872 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.704778    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-oauth-serving-cert\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.704872 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.704864    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-trusted-ca-bundle\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.706347 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.706320    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-console-serving-cert\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.706461 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.706433    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-console-oauth-config\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.712987 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.712964    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbdtc\" (UniqueName: \"kubernetes.io/projected/5cae9eea-5aeb-4330-9d8f-3e5d6fa20887-kube-api-access-vbdtc\") pod \"console-9695b8856-x4klq\" (UID: \"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887\") " pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.821396 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.821287    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:20.959566 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.959498    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9695b8856-x4klq"]
May 20 15:38:20.962268 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:38:20.962241    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cae9eea_5aeb_4330_9d8f_3e5d6fa20887.slice/crio-f486c51c8721865776a71fc1336cf22d69eb8cd704c1949cb03ff512a3d5a2ba WatchSource:0}: Error finding container f486c51c8721865776a71fc1336cf22d69eb8cd704c1949cb03ff512a3d5a2ba: Status 404 returned error can't find the container with id f486c51c8721865776a71fc1336cf22d69eb8cd704c1949cb03ff512a3d5a2ba
May 20 15:38:20.964055 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:20.964038    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:38:21.613457 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:21.613414    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9695b8856-x4klq" event={"ID":"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887","Type":"ContainerStarted","Data":"e29e2392a7564809701637874fc81c05531d5aaacceef391edbe1ead048e3966"}
May 20 15:38:21.613457 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:21.613453    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9695b8856-x4klq" event={"ID":"5cae9eea-5aeb-4330-9d8f-3e5d6fa20887","Type":"ContainerStarted","Data":"f486c51c8721865776a71fc1336cf22d69eb8cd704c1949cb03ff512a3d5a2ba"}
May 20 15:38:21.637668 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:21.637617    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-9695b8856-x4klq" podStartSLOduration=1.637578123 podStartE2EDuration="1.637578123s" podCreationTimestamp="2026-05-20 15:38:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:38:21.63497848 +0000 UTC m=+391.339072070" watchObservedRunningTime="2026-05-20 15:38:21.637578123 +0000 UTC m=+391.341671713"
May 20 15:38:25.115362 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.115324    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"]
May 20 15:38:25.119004 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.118980    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:25.121787 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.121762    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 15:38:25.122951 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.122932    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 15:38:25.123081 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.122967    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g25pr\""
May 20 15:38:25.127333 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.127303    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"]
May 20 15:38:25.138694 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.138653    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxlb\" (UniqueName: \"kubernetes.io/projected/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-kube-api-access-phxlb\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:25.138867 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.138765    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:25.138867 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.138817    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:25.239842 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.239796    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:25.239992 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.239857    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-phxlb\" (UniqueName: \"kubernetes.io/projected/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-kube-api-access-phxlb\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:25.239992 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.239904    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:25.240269 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.240252    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:25.240313 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.240251    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:25.249383 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.249343    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxlb\" (UniqueName: \"kubernetes.io/projected/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-kube-api-access-phxlb\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:25.430089 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.429985    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:25.565574 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.565533    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"]
May 20 15:38:25.567715 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:38:25.567685    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd79fe4b_6c3f_4e92_8d90_b49af59694f0.slice/crio-0bfac069dcbb016b610a653410a99a4bdce0c27764bce0075f8d4a608b7fadc2 WatchSource:0}: Error finding container 0bfac069dcbb016b610a653410a99a4bdce0c27764bce0075f8d4a608b7fadc2: Status 404 returned error can't find the container with id 0bfac069dcbb016b610a653410a99a4bdce0c27764bce0075f8d4a608b7fadc2
May 20 15:38:25.625732 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:25.625698    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7" event={"ID":"cd79fe4b-6c3f-4e92-8d90-b49af59694f0","Type":"ContainerStarted","Data":"0bfac069dcbb016b610a653410a99a4bdce0c27764bce0075f8d4a608b7fadc2"}
May 20 15:38:30.645427 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:30.645385    2567 generic.go:358] "Generic (PLEG): container finished" podID="cd79fe4b-6c3f-4e92-8d90-b49af59694f0" containerID="f1591e106faa3feb48fdf7b10aba4806c9d21cf09147e5f597496d71eff011d8" exitCode=0
May 20 15:38:30.645956 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:30.645466    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7" event={"ID":"cd79fe4b-6c3f-4e92-8d90-b49af59694f0","Type":"ContainerDied","Data":"f1591e106faa3feb48fdf7b10aba4806c9d21cf09147e5f597496d71eff011d8"}
May 20 15:38:30.822103 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:30.822067    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:30.822287 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:30.822229    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:30.827071 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:30.827048    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:31.653840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:31.653809    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9695b8856-x4klq"
May 20 15:38:31.706943 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:31.706911    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59bfb67c-gzjbj"]
May 20 15:38:32.653618 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:32.653564    2567 generic.go:358] "Generic (PLEG): container finished" podID="cd79fe4b-6c3f-4e92-8d90-b49af59694f0" containerID="3d84213948d414f6b3c918b5854a2ae6a0e81c9de4e84dd91128c41a81ce15c4" exitCode=0
May 20 15:38:32.653809 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:32.653669    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7" event={"ID":"cd79fe4b-6c3f-4e92-8d90-b49af59694f0","Type":"ContainerDied","Data":"3d84213948d414f6b3c918b5854a2ae6a0e81c9de4e84dd91128c41a81ce15c4"}
May 20 15:38:38.674398 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:38.674360    2567 generic.go:358] "Generic (PLEG): container finished" podID="cd79fe4b-6c3f-4e92-8d90-b49af59694f0" containerID="3cc90dd5234025d681ac50328a5351f02434685673024f66f7dfb94a6446c095" exitCode=0
May 20 15:38:38.674794 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:38.674436    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7" event={"ID":"cd79fe4b-6c3f-4e92-8d90-b49af59694f0","Type":"ContainerDied","Data":"3cc90dd5234025d681ac50328a5351f02434685673024f66f7dfb94a6446c095"}
May 20 15:38:39.802501 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:39.802471    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:39.868688 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:39.868655    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-bundle\") pod \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") "
May 20 15:38:39.868864 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:39.868722    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-util\") pod \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") "
May 20 15:38:39.868864 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:39.868752    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phxlb\" (UniqueName: \"kubernetes.io/projected/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-kube-api-access-phxlb\") pod \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\" (UID: \"cd79fe4b-6c3f-4e92-8d90-b49af59694f0\") "
May 20 15:38:39.869359 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:39.869316    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-bundle" (OuterVolumeSpecName: "bundle") pod "cd79fe4b-6c3f-4e92-8d90-b49af59694f0" (UID: "cd79fe4b-6c3f-4e92-8d90-b49af59694f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:38:39.871095 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:39.871064    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-kube-api-access-phxlb" (OuterVolumeSpecName: "kube-api-access-phxlb") pod "cd79fe4b-6c3f-4e92-8d90-b49af59694f0" (UID: "cd79fe4b-6c3f-4e92-8d90-b49af59694f0"). InnerVolumeSpecName "kube-api-access-phxlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:38:39.873819 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:39.873794    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-util" (OuterVolumeSpecName: "util") pod "cd79fe4b-6c3f-4e92-8d90-b49af59694f0" (UID: "cd79fe4b-6c3f-4e92-8d90-b49af59694f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:38:39.969925 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:39.969840    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:38:39.969925 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:39.969870    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-util\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:38:39.969925 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:39.969879    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phxlb\" (UniqueName: \"kubernetes.io/projected/cd79fe4b-6c3f-4e92-8d90-b49af59694f0-kube-api-access-phxlb\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:38:40.682768 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:40.682731    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7" event={"ID":"cd79fe4b-6c3f-4e92-8d90-b49af59694f0","Type":"ContainerDied","Data":"0bfac069dcbb016b610a653410a99a4bdce0c27764bce0075f8d4a608b7fadc2"}
May 20 15:38:40.682768 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:40.682770    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bfac069dcbb016b610a653410a99a4bdce0c27764bce0075f8d4a608b7fadc2"
May 20 15:38:40.682982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:40.682776    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cfrsb7"
May 20 15:38:47.108970 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.108931    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"]
May 20 15:38:47.109427 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.109283    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd79fe4b-6c3f-4e92-8d90-b49af59694f0" containerName="pull"
May 20 15:38:47.109427 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.109294    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd79fe4b-6c3f-4e92-8d90-b49af59694f0" containerName="pull"
May 20 15:38:47.109427 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.109318    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd79fe4b-6c3f-4e92-8d90-b49af59694f0" containerName="util"
May 20 15:38:47.109427 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.109324    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd79fe4b-6c3f-4e92-8d90-b49af59694f0" containerName="util"
May 20 15:38:47.109427 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.109333    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd79fe4b-6c3f-4e92-8d90-b49af59694f0" containerName="extract"
May 20 15:38:47.109427 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.109338    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd79fe4b-6c3f-4e92-8d90-b49af59694f0" containerName="extract"
May 20 15:38:47.109427 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.109387    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd79fe4b-6c3f-4e92-8d90-b49af59694f0" containerName="extract"
May 20 15:38:47.113106 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.113084    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"
May 20 15:38:47.115944 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.115924    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 20 15:38:47.116305 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.116285    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-b6mc9\""
May 20 15:38:47.116570 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.116552    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 20 15:38:47.116651 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.116578    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 20 15:38:47.140749 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.140712    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"]
May 20 15:38:47.235667 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.235629    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/55e543ce-e469-4df7-adb6-82ab341d2185-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-9wssl\" (UID: \"55e543ce-e469-4df7-adb6-82ab341d2185\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"
May 20 15:38:47.235757 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.235691    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjv5g\" (UniqueName: \"kubernetes.io/projected/55e543ce-e469-4df7-adb6-82ab341d2185-kube-api-access-xjv5g\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-9wssl\" (UID: \"55e543ce-e469-4df7-adb6-82ab341d2185\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"
May 20 15:38:47.336765 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.336723    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/55e543ce-e469-4df7-adb6-82ab341d2185-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-9wssl\" (UID: \"55e543ce-e469-4df7-adb6-82ab341d2185\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"
May 20 15:38:47.336926 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.336775    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjv5g\" (UniqueName: \"kubernetes.io/projected/55e543ce-e469-4df7-adb6-82ab341d2185-kube-api-access-xjv5g\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-9wssl\" (UID: \"55e543ce-e469-4df7-adb6-82ab341d2185\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"
May 20 15:38:47.339209 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.339188    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/55e543ce-e469-4df7-adb6-82ab341d2185-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-9wssl\" (UID: \"55e543ce-e469-4df7-adb6-82ab341d2185\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"
May 20 15:38:47.345376 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.345350    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjv5g\" (UniqueName: \"kubernetes.io/projected/55e543ce-e469-4df7-adb6-82ab341d2185-kube-api-access-xjv5g\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-9wssl\" (UID: \"55e543ce-e469-4df7-adb6-82ab341d2185\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"
May 20 15:38:47.423431 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.423332    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"
May 20 15:38:47.576026 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.575989    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"]
May 20 15:38:47.580608 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:38:47.580555    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e543ce_e469_4df7_adb6_82ab341d2185.slice/crio-a897b4ae7d747d033e6b08013d8a0a14bfb08703bff6d766e05d44bc66d3ce9a WatchSource:0}: Error finding container a897b4ae7d747d033e6b08013d8a0a14bfb08703bff6d766e05d44bc66d3ce9a: Status 404 returned error can't find the container with id a897b4ae7d747d033e6b08013d8a0a14bfb08703bff6d766e05d44bc66d3ce9a
May 20 15:38:47.704410 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:47.704318    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl" event={"ID":"55e543ce-e469-4df7-adb6-82ab341d2185","Type":"ContainerStarted","Data":"a897b4ae7d747d033e6b08013d8a0a14bfb08703bff6d766e05d44bc66d3ce9a"}
May 20 15:38:52.722901 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:52.722857    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl" event={"ID":"55e543ce-e469-4df7-adb6-82ab341d2185","Type":"ContainerStarted","Data":"946cd34414f66c4e18d85bce00252c04d9b46b70904a8010710bb64a5aa97ee4"}
May 20 15:38:52.723305 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:52.722996    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"
May 20 15:38:52.742902 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:52.742745    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl" podStartSLOduration=0.770207418 podStartE2EDuration="5.74272952s" podCreationTimestamp="2026-05-20 15:38:47 +0000 UTC" firstStartedPulling="2026-05-20 15:38:47.582352534 +0000 UTC m=+417.286446103" lastFinishedPulling="2026-05-20 15:38:52.554874634 +0000 UTC m=+422.258968205" observedRunningTime="2026-05-20 15:38:52.741264179 +0000 UTC m=+422.445357770" watchObservedRunningTime="2026-05-20 15:38:52.74272952 +0000 UTC m=+422.446823110"
May 20 15:38:53.076771 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.076729    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-fxr4m"]
May 20 15:38:53.079330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.079307    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:53.082185 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.082162    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 20 15:38:53.082294 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.082189    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 20 15:38:53.082294 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.082162    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hsshr\""
May 20 15:38:53.088729 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.088695    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-fxr4m"]
May 20 15:38:53.190323 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.190285    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-cabundle0\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:53.190507 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.190327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pljm2\" (UniqueName: \"kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-kube-api-access-pljm2\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:53.190507 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.190417    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:53.291744 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.291695    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-cabundle0\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:53.291945 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.291751    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pljm2\" (UniqueName: \"kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-kube-api-access-pljm2\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:53.291945 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.291834    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:53.292058 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.291974    2567 secret.go:281] references non-existent secret key: ca.crt
May 20 15:38:53.292058 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.291990    2567 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 15:38:53.292058 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.292001    2567 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-fxr4m: references non-existent secret key: ca.crt
May 20 15:38:53.292200 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.292062    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates podName:b9df9cfe-c9da-4ee5-bc1e-cf36729a013a nodeName:}" failed. No retries permitted until 2026-05-20 15:38:53.79204193 +0000 UTC m=+423.496135503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates") pod "keda-operator-ffbb595cb-fxr4m" (UID: "b9df9cfe-c9da-4ee5-bc1e-cf36729a013a") : references non-existent secret key: ca.crt
May 20 15:38:53.292359 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.292339    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-cabundle0\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:53.304382 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.304349    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pljm2\" (UniqueName: \"kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-kube-api-access-pljm2\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:53.435146 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.435106    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"]
May 20 15:38:53.437404 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.437384    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:53.440093 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.440071    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 20 15:38:53.449929 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.449900    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"]
May 20 15:38:53.595150 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.595104    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/357f2b55-919e-449d-b24a-8b5ac92b8982-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:53.595354 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.595166    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:53.595354 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.595209    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwm26\" (UniqueName: \"kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-kube-api-access-kwm26\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:53.696188 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.696106    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/357f2b55-919e-449d-b24a-8b5ac92b8982-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:53.696188 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.696175    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:53.696365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.696226    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kwm26\" (UniqueName: \"kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-kube-api-access-kwm26\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:53.696365 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.696323    2567 secret.go:281] references non-existent secret key: tls.crt
May 20 15:38:53.696365 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.696341    2567 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 20 15:38:53.696365 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.696361    2567 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt: references non-existent secret key: tls.crt
May 20 15:38:53.696514 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.696415    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates podName:357f2b55-919e-449d-b24a-8b5ac92b8982 nodeName:}" failed. No retries permitted until 2026-05-20 15:38:54.196400694 +0000 UTC m=+423.900494263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates") pod "keda-metrics-apiserver-7c9f485588-sfrdt" (UID: "357f2b55-919e-449d-b24a-8b5ac92b8982") : references non-existent secret key: tls.crt
May 20 15:38:53.696617 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.696566    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/357f2b55-919e-449d-b24a-8b5ac92b8982-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:53.707347 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.707322    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwm26\" (UniqueName: \"kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-kube-api-access-kwm26\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:53.735555 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.735517    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-xmv59"]
May 20 15:38:53.737813 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.737792    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-xmv59"
May 20 15:38:53.740474 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.740452    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
May 20 15:38:53.747465 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.747432    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-xmv59"]
May 20 15:38:53.796949 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.796909    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:53.797160 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.797063    2567 secret.go:281] references non-existent secret key: ca.crt
May 20 15:38:53.797160 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.797084    2567 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 15:38:53.797160 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.797093    2567 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-fxr4m: references non-existent secret key: ca.crt
May 20 15:38:53.797160 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:53.797143    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates podName:b9df9cfe-c9da-4ee5-bc1e-cf36729a013a nodeName:}" failed. No retries permitted until 2026-05-20 15:38:54.797128573 +0000 UTC m=+424.501222146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates") pod "keda-operator-ffbb595cb-fxr4m" (UID: "b9df9cfe-c9da-4ee5-bc1e-cf36729a013a") : references non-existent secret key: ca.crt
May 20 15:38:53.897487 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.897452    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2079ee92-468e-4908-a6bf-888efe39d1a8-certificates\") pod \"keda-admission-cf49989db-xmv59\" (UID: \"2079ee92-468e-4908-a6bf-888efe39d1a8\") " pod="openshift-keda/keda-admission-cf49989db-xmv59"
May 20 15:38:53.897682 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.897539    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl8nj\" (UniqueName: \"kubernetes.io/projected/2079ee92-468e-4908-a6bf-888efe39d1a8-kube-api-access-zl8nj\") pod \"keda-admission-cf49989db-xmv59\" (UID: \"2079ee92-468e-4908-a6bf-888efe39d1a8\") " pod="openshift-keda/keda-admission-cf49989db-xmv59"
May 20 15:38:53.998345 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.998240    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zl8nj\" (UniqueName: \"kubernetes.io/projected/2079ee92-468e-4908-a6bf-888efe39d1a8-kube-api-access-zl8nj\") pod \"keda-admission-cf49989db-xmv59\" (UID: \"2079ee92-468e-4908-a6bf-888efe39d1a8\") " pod="openshift-keda/keda-admission-cf49989db-xmv59"
May 20 15:38:53.998345 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:53.998337    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2079ee92-468e-4908-a6bf-888efe39d1a8-certificates\") pod \"keda-admission-cf49989db-xmv59\" (UID: \"2079ee92-468e-4908-a6bf-888efe39d1a8\") " pod="openshift-keda/keda-admission-cf49989db-xmv59"
May 20 15:38:54.001994 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:54.001967    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2079ee92-468e-4908-a6bf-888efe39d1a8-certificates\") pod \"keda-admission-cf49989db-xmv59\" (UID: \"2079ee92-468e-4908-a6bf-888efe39d1a8\") " pod="openshift-keda/keda-admission-cf49989db-xmv59"
May 20 15:38:54.007732 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:54.007699    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl8nj\" (UniqueName: \"kubernetes.io/projected/2079ee92-468e-4908-a6bf-888efe39d1a8-kube-api-access-zl8nj\") pod \"keda-admission-cf49989db-xmv59\" (UID: \"2079ee92-468e-4908-a6bf-888efe39d1a8\") " pod="openshift-keda/keda-admission-cf49989db-xmv59"
May 20 15:38:54.049723 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:54.049682    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-xmv59"
May 20 15:38:54.196565 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:54.196537    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-xmv59"]
May 20 15:38:54.199348 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:38:54.199323    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2079ee92_468e_4908_a6bf_888efe39d1a8.slice/crio-fa5e8d3c8c15ac59ab4794f26fda9da9150bdee117efaa56540f44ee0ce7ac32 WatchSource:0}: Error finding container fa5e8d3c8c15ac59ab4794f26fda9da9150bdee117efaa56540f44ee0ce7ac32: Status 404 returned error can't find the container with id fa5e8d3c8c15ac59ab4794f26fda9da9150bdee117efaa56540f44ee0ce7ac32
May 20 15:38:54.200452 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:54.200426    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:54.200603 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:54.200569    2567 secret.go:281] references non-existent secret key: tls.crt
May 20 15:38:54.200679 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:54.200608    2567 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 20 15:38:54.200679 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:54.200630    2567 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt: references non-existent secret key: tls.crt
May 20 15:38:54.200679 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:54.200680    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates podName:357f2b55-919e-449d-b24a-8b5ac92b8982 nodeName:}" failed. No retries permitted until 2026-05-20 15:38:55.200663297 +0000 UTC m=+424.904756868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates") pod "keda-metrics-apiserver-7c9f485588-sfrdt" (UID: "357f2b55-919e-449d-b24a-8b5ac92b8982") : references non-existent secret key: tls.crt
May 20 15:38:54.730744 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:54.730702    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-xmv59" event={"ID":"2079ee92-468e-4908-a6bf-888efe39d1a8","Type":"ContainerStarted","Data":"fa5e8d3c8c15ac59ab4794f26fda9da9150bdee117efaa56540f44ee0ce7ac32"}
May 20 15:38:54.807191 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:54.807149    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:54.807565 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:54.807302    2567 secret.go:281] references non-existent secret key: ca.crt
May 20 15:38:54.807565 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:54.807318    2567 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 20 15:38:54.807565 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:54.807327    2567 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-fxr4m: references non-existent secret key: ca.crt
May 20 15:38:54.807565 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:54.807382    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates podName:b9df9cfe-c9da-4ee5-bc1e-cf36729a013a nodeName:}" failed. No retries permitted until 2026-05-20 15:38:56.807365602 +0000 UTC m=+426.511459187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates") pod "keda-operator-ffbb595cb-fxr4m" (UID: "b9df9cfe-c9da-4ee5-bc1e-cf36729a013a") : references non-existent secret key: ca.crt
May 20 15:38:55.211225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:55.211177    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:55.211408 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:55.211338    2567 secret.go:281] references non-existent secret key: tls.crt
May 20 15:38:55.211408 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:55.211358    2567 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 20 15:38:55.211408 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:55.211382    2567 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt: references non-existent secret key: tls.crt
May 20 15:38:55.211683 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:55.211452    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates podName:357f2b55-919e-449d-b24a-8b5ac92b8982 nodeName:}" failed. No retries permitted until 2026-05-20 15:38:57.211436741 +0000 UTC m=+426.915530314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates") pod "keda-metrics-apiserver-7c9f485588-sfrdt" (UID: "357f2b55-919e-449d-b24a-8b5ac92b8982") : references non-existent secret key: tls.crt
May 20 15:38:56.728700 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:56.728575    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-59bfb67c-gzjbj" podUID="604888dc-fa6b-4598-a157-919087886059" containerName="console" containerID="cri-o://115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370" gracePeriod=15
May 20 15:38:56.739335 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:56.739217    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-xmv59" event={"ID":"2079ee92-468e-4908-a6bf-888efe39d1a8","Type":"ContainerStarted","Data":"b681e2f6f5c661e80da615d403259756efee4dab5fb40579d741c6bb06562f02"}
May 20 15:38:56.739420 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:56.739378    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-xmv59"
May 20 15:38:56.758013 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:56.757957    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-xmv59" podStartSLOduration=1.5457946009999999 podStartE2EDuration="3.757942831s" podCreationTimestamp="2026-05-20 15:38:53 +0000 UTC" firstStartedPulling="2026-05-20 15:38:54.200543125 +0000 UTC m=+423.904636695" lastFinishedPulling="2026-05-20 15:38:56.412691357 +0000 UTC m=+426.116784925" observedRunningTime="2026-05-20 15:38:56.756364645 +0000 UTC m=+426.460458237" watchObservedRunningTime="2026-05-20 15:38:56.757942831 +0000 UTC m=+426.462036421"
May 20 15:38:56.828755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:56.828710    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:56.831340 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:56.831322    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b9df9cfe-c9da-4ee5-bc1e-cf36729a013a-certificates\") pod \"keda-operator-ffbb595cb-fxr4m\" (UID: \"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a\") " pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:56.970233 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:56.970209    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59bfb67c-gzjbj_604888dc-fa6b-4598-a157-919087886059/console/0.log"
May 20 15:38:56.970366 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:56.970273    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:38:56.990153 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:56.990066    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:38:57.124135 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.124105    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-fxr4m"]
May 20 15:38:57.125972 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:38:57.125941    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9df9cfe_c9da_4ee5_bc1e_cf36729a013a.slice/crio-0c562d5b06afc3a5aef859b5e7d378a7d185cba4c4c16708366bb70b4e9d5fe2 WatchSource:0}: Error finding container 0c562d5b06afc3a5aef859b5e7d378a7d185cba4c4c16708366bb70b4e9d5fe2: Status 404 returned error can't find the container with id 0c562d5b06afc3a5aef859b5e7d378a7d185cba4c4c16708366bb70b4e9d5fe2
May 20 15:38:57.131870 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.131848    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-trusted-ca-bundle\") pod \"604888dc-fa6b-4598-a157-919087886059\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") "
May 20 15:38:57.131964 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.131896    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-serving-cert\") pod \"604888dc-fa6b-4598-a157-919087886059\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") "
May 20 15:38:57.131964 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.131920    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-service-ca\") pod \"604888dc-fa6b-4598-a157-919087886059\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") "
May 20 15:38:57.131964 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.131938    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q5l7\" (UniqueName: \"kubernetes.io/projected/604888dc-fa6b-4598-a157-919087886059-kube-api-access-2q5l7\") pod \"604888dc-fa6b-4598-a157-919087886059\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") "
May 20 15:38:57.131964 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.131955    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-oauth-serving-cert\") pod \"604888dc-fa6b-4598-a157-919087886059\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") "
May 20 15:38:57.132160 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.131978    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-console-config\") pod \"604888dc-fa6b-4598-a157-919087886059\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") "
May 20 15:38:57.132160 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.132062    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-oauth-config\") pod \"604888dc-fa6b-4598-a157-919087886059\" (UID: \"604888dc-fa6b-4598-a157-919087886059\") "
May 20 15:38:57.132427 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.132391    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "604888dc-fa6b-4598-a157-919087886059" (UID: "604888dc-fa6b-4598-a157-919087886059"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:38:57.132522 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.132404    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "604888dc-fa6b-4598-a157-919087886059" (UID: "604888dc-fa6b-4598-a157-919087886059"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:38:57.132522 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.132404    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-service-ca" (OuterVolumeSpecName: "service-ca") pod "604888dc-fa6b-4598-a157-919087886059" (UID: "604888dc-fa6b-4598-a157-919087886059"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:38:57.132522 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.132494    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-trusted-ca-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:38:57.132522 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.132514    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-service-ca\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:38:57.132784 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.132529    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-oauth-serving-cert\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:38:57.133466 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.133439    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-console-config" (OuterVolumeSpecName: "console-config") pod "604888dc-fa6b-4598-a157-919087886059" (UID: "604888dc-fa6b-4598-a157-919087886059"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 20 15:38:57.134530 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.134510    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604888dc-fa6b-4598-a157-919087886059-kube-api-access-2q5l7" (OuterVolumeSpecName: "kube-api-access-2q5l7") pod "604888dc-fa6b-4598-a157-919087886059" (UID: "604888dc-fa6b-4598-a157-919087886059"). InnerVolumeSpecName "kube-api-access-2q5l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:38:57.134957 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.134935    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "604888dc-fa6b-4598-a157-919087886059" (UID: "604888dc-fa6b-4598-a157-919087886059"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:38:57.135036 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.135017    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "604888dc-fa6b-4598-a157-919087886059" (UID: "604888dc-fa6b-4598-a157-919087886059"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:38:57.233731 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.233682    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:57.233903 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.233819    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/604888dc-fa6b-4598-a157-919087886059-console-config\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:38:57.233903 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.233837    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-oauth-config\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:38:57.233903 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.233850    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/604888dc-fa6b-4598-a157-919087886059-console-serving-cert\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:38:57.233903 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.233863    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q5l7\" (UniqueName: \"kubernetes.io/projected/604888dc-fa6b-4598-a157-919087886059-kube-api-access-2q5l7\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:38:57.236627 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.236568    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/357f2b55-919e-449d-b24a-8b5ac92b8982-certificates\") pod \"keda-metrics-apiserver-7c9f485588-sfrdt\" (UID: \"357f2b55-919e-449d-b24a-8b5ac92b8982\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:57.348779 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.348747    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:38:57.482723 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.482697    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"]
May 20 15:38:57.484919 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:38:57.484897    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357f2b55_919e_449d_b24a_8b5ac92b8982.slice/crio-ddcfc128a7370ffd05eb3737f6f6d6aa414f0478c1bca8d7fa744e69b081cb3c WatchSource:0}: Error finding container ddcfc128a7370ffd05eb3737f6f6d6aa414f0478c1bca8d7fa744e69b081cb3c: Status 404 returned error can't find the container with id ddcfc128a7370ffd05eb3737f6f6d6aa414f0478c1bca8d7fa744e69b081cb3c
May 20 15:38:57.746264 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.746161    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt" event={"ID":"357f2b55-919e-449d-b24a-8b5ac92b8982","Type":"ContainerStarted","Data":"ddcfc128a7370ffd05eb3737f6f6d6aa414f0478c1bca8d7fa744e69b081cb3c"}
May 20 15:38:57.747790 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.747770    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59bfb67c-gzjbj_604888dc-fa6b-4598-a157-919087886059/console/0.log"
May 20 15:38:57.747906 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.747805    2567 generic.go:358] "Generic (PLEG): container finished" podID="604888dc-fa6b-4598-a157-919087886059" containerID="115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370" exitCode=2
May 20 15:38:57.747906 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.747829    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bfb67c-gzjbj" event={"ID":"604888dc-fa6b-4598-a157-919087886059","Type":"ContainerDied","Data":"115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370"}
May 20 15:38:57.747906 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.747865    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bfb67c-gzjbj" event={"ID":"604888dc-fa6b-4598-a157-919087886059","Type":"ContainerDied","Data":"de163dc8af249f8f053cec6d1c073d0f78568343fcb6689f1926354425b48598"}
May 20 15:38:57.747906 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.747882    2567 scope.go:117] "RemoveContainer" containerID="115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370"
May 20 15:38:57.748113 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.747905    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bfb67c-gzjbj"
May 20 15:38:57.749001 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.748977    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-fxr4m" event={"ID":"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a","Type":"ContainerStarted","Data":"0c562d5b06afc3a5aef859b5e7d378a7d185cba4c4c16708366bb70b4e9d5fe2"}
May 20 15:38:57.756427 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.756406    2567 scope.go:117] "RemoveContainer" containerID="115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370"
May 20 15:38:57.756721 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:38:57.756701    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370\": container with ID starting with 115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370 not found: ID does not exist" containerID="115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370"
May 20 15:38:57.756796 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.756734    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370"} err="failed to get container status \"115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370\": rpc error: code = NotFound desc = could not find container \"115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370\": container with ID starting with 115efc0d3b33cc6c77b1b500e448757d6f9a329e68ae5d7fa542b8d7465bd370 not found: ID does not exist"
May 20 15:38:57.769136 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.769112    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59bfb67c-gzjbj"]
May 20 15:38:57.773037 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:57.773011    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59bfb67c-gzjbj"]
May 20 15:38:59.057850 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:38:59.057814    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="604888dc-fa6b-4598-a157-919087886059" path="/var/lib/kubelet/pods/604888dc-fa6b-4598-a157-919087886059/volumes"
May 20 15:39:02.771359 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:02.771320    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-fxr4m" event={"ID":"b9df9cfe-c9da-4ee5-bc1e-cf36729a013a","Type":"ContainerStarted","Data":"55e51f052e8c8dfdecfe73e0d570978861abffb76407865b73a9455af95237d1"}
May 20 15:39:02.771876 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:02.771390    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:39:02.772626 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:02.772577    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt" event={"ID":"357f2b55-919e-449d-b24a-8b5ac92b8982","Type":"ContainerStarted","Data":"f126249ffbf91c99f93e3c60713f269d92dedbf6ba51f0b4553410dfeb1ab2db"}
May 20 15:39:02.772767 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:02.772752    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:39:02.787958 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:02.787903    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-fxr4m" podStartSLOduration=5.201784309 podStartE2EDuration="9.787886542s" podCreationTimestamp="2026-05-20 15:38:53 +0000 UTC" firstStartedPulling="2026-05-20 15:38:57.127290508 +0000 UTC m=+426.831384090" lastFinishedPulling="2026-05-20 15:39:01.713392756 +0000 UTC m=+431.417486323" observedRunningTime="2026-05-20 15:39:02.786506179 +0000 UTC m=+432.490599770" watchObservedRunningTime="2026-05-20 15:39:02.787886542 +0000 UTC m=+432.491980136"
May 20 15:39:02.803867 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:02.803812    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt" podStartSLOduration=5.580149361 podStartE2EDuration="9.803797098s" podCreationTimestamp="2026-05-20 15:38:53 +0000 UTC" firstStartedPulling="2026-05-20 15:38:57.486662055 +0000 UTC m=+427.190755623" lastFinishedPulling="2026-05-20 15:39:01.710309775 +0000 UTC m=+431.414403360" observedRunningTime="2026-05-20 15:39:02.802089433 +0000 UTC m=+432.506183022" watchObservedRunningTime="2026-05-20 15:39:02.803797098 +0000 UTC m=+432.507890735"
May 20 15:39:13.728511 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:13.728479    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-9wssl"
May 20 15:39:13.779999 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:13.779972    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-sfrdt"
May 20 15:39:17.751706 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:17.751660    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-xmv59"
May 20 15:39:23.778635 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:23.778574    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-fxr4m"
May 20 15:39:58.855312 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.855270    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-d2s9d"]
May 20 15:39:58.856016 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.855845    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="604888dc-fa6b-4598-a157-919087886059" containerName="console"
May 20 15:39:58.856016 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.855866    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="604888dc-fa6b-4598-a157-919087886059" containerName="console"
May 20 15:39:58.856016 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.855969    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="604888dc-fa6b-4598-a157-919087886059" containerName="console"
May 20 15:39:58.858639 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.858617    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:39:58.861756 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.861734    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 20 15:39:58.861885 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.861756    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 20 15:39:58.862914 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.862897    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-x8cbn\""
May 20 15:39:58.862970 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.862896    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 20 15:39:58.872112 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.872083    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-d2s9d"]
May 20 15:39:58.891472 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.891436    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-8677dc7484-bsvzf"]
May 20 15:39:58.893701 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.893672    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:39:58.896610 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.896566    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 20 15:39:58.896735 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.896660    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-mvg2p\""
May 20 15:39:58.901467 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.901420    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-8677dc7484-bsvzf"]
May 20 15:39:58.912106 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.912077    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-znl9n"]
May 20 15:39:58.914673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.914649    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-znl9n"
May 20 15:39:58.917272 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.917249    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-7wj7m\""
May 20 15:39:58.917391 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.917249    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 20 15:39:58.925070 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.925042    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-znl9n"]
May 20 15:39:58.956912 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.956864    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2lc\" (UniqueName: \"kubernetes.io/projected/5c200523-580a-4852-9cff-0318b950ca82-kube-api-access-pf2lc\") pod \"kserve-controller-manager-87b4b9896-d2s9d\" (UID: \"5c200523-580a-4852-9cff-0318b950ca82\") " pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:39:58.957103 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:58.957030    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c200523-580a-4852-9cff-0318b950ca82-cert\") pod \"kserve-controller-manager-87b4b9896-d2s9d\" (UID: \"5c200523-580a-4852-9cff-0318b950ca82\") " pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:39:59.057487 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.057445    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f85282ae-679b-472b-b395-88f43b596eb0-cert\") pod \"llmisvc-controller-manager-8677dc7484-bsvzf\" (UID: \"f85282ae-679b-472b-b395-88f43b596eb0\") " pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:39:59.057487 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.057482    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/891539dc-9b96-4d97-a03a-fa8826801d9b-data\") pod \"seaweedfs-79f457656b-znl9n\" (UID: \"891539dc-9b96-4d97-a03a-fa8826801d9b\") " pod="kserve/seaweedfs-79f457656b-znl9n"
May 20 15:39:59.057761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.057521    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdgg\" (UniqueName: \"kubernetes.io/projected/891539dc-9b96-4d97-a03a-fa8826801d9b-kube-api-access-rzdgg\") pod \"seaweedfs-79f457656b-znl9n\" (UID: \"891539dc-9b96-4d97-a03a-fa8826801d9b\") " pod="kserve/seaweedfs-79f457656b-znl9n"
May 20 15:39:59.057761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.057567    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c200523-580a-4852-9cff-0318b950ca82-cert\") pod \"kserve-controller-manager-87b4b9896-d2s9d\" (UID: \"5c200523-580a-4852-9cff-0318b950ca82\") " pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:39:59.057761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.057619    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2lc\" (UniqueName: \"kubernetes.io/projected/5c200523-580a-4852-9cff-0318b950ca82-kube-api-access-pf2lc\") pod \"kserve-controller-manager-87b4b9896-d2s9d\" (UID: \"5c200523-580a-4852-9cff-0318b950ca82\") " pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:39:59.057761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.057679    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kql7v\" (UniqueName: \"kubernetes.io/projected/f85282ae-679b-472b-b395-88f43b596eb0-kube-api-access-kql7v\") pod \"llmisvc-controller-manager-8677dc7484-bsvzf\" (UID: \"f85282ae-679b-472b-b395-88f43b596eb0\") " pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:39:59.057761 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:39:59.057733    2567 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
May 20 15:39:59.057927 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:39:59.057812    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c200523-580a-4852-9cff-0318b950ca82-cert podName:5c200523-580a-4852-9cff-0318b950ca82 nodeName:}" failed. No retries permitted until 2026-05-20 15:39:59.557794783 +0000 UTC m=+489.261888357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5c200523-580a-4852-9cff-0318b950ca82-cert") pod "kserve-controller-manager-87b4b9896-d2s9d" (UID: "5c200523-580a-4852-9cff-0318b950ca82") : secret "kserve-webhook-server-cert" not found
May 20 15:39:59.069704 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.069678    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2lc\" (UniqueName: \"kubernetes.io/projected/5c200523-580a-4852-9cff-0318b950ca82-kube-api-access-pf2lc\") pod \"kserve-controller-manager-87b4b9896-d2s9d\" (UID: \"5c200523-580a-4852-9cff-0318b950ca82\") " pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:39:59.158374 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.158279    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f85282ae-679b-472b-b395-88f43b596eb0-cert\") pod \"llmisvc-controller-manager-8677dc7484-bsvzf\" (UID: \"f85282ae-679b-472b-b395-88f43b596eb0\") " pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:39:59.158374 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.158318    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/891539dc-9b96-4d97-a03a-fa8826801d9b-data\") pod \"seaweedfs-79f457656b-znl9n\" (UID: \"891539dc-9b96-4d97-a03a-fa8826801d9b\") " pod="kserve/seaweedfs-79f457656b-znl9n"
May 20 15:39:59.158667 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:39:59.158445    2567 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
May 20 15:39:59.158667 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:39:59.158529    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85282ae-679b-472b-b395-88f43b596eb0-cert podName:f85282ae-679b-472b-b395-88f43b596eb0 nodeName:}" failed. No retries permitted until 2026-05-20 15:39:59.658497466 +0000 UTC m=+489.362591037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f85282ae-679b-472b-b395-88f43b596eb0-cert") pod "llmisvc-controller-manager-8677dc7484-bsvzf" (UID: "f85282ae-679b-472b-b395-88f43b596eb0") : secret "llmisvc-webhook-server-cert" not found
May 20 15:39:59.158667 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.158577    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdgg\" (UniqueName: \"kubernetes.io/projected/891539dc-9b96-4d97-a03a-fa8826801d9b-kube-api-access-rzdgg\") pod \"seaweedfs-79f457656b-znl9n\" (UID: \"891539dc-9b96-4d97-a03a-fa8826801d9b\") " pod="kserve/seaweedfs-79f457656b-znl9n"
May 20 15:39:59.158827 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.158770    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/891539dc-9b96-4d97-a03a-fa8826801d9b-data\") pod \"seaweedfs-79f457656b-znl9n\" (UID: \"891539dc-9b96-4d97-a03a-fa8826801d9b\") " pod="kserve/seaweedfs-79f457656b-znl9n"
May 20 15:39:59.158888 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.158837    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kql7v\" (UniqueName: \"kubernetes.io/projected/f85282ae-679b-472b-b395-88f43b596eb0-kube-api-access-kql7v\") pod \"llmisvc-controller-manager-8677dc7484-bsvzf\" (UID: \"f85282ae-679b-472b-b395-88f43b596eb0\") " pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:39:59.168063 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.168025    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kql7v\" (UniqueName: \"kubernetes.io/projected/f85282ae-679b-472b-b395-88f43b596eb0-kube-api-access-kql7v\") pod \"llmisvc-controller-manager-8677dc7484-bsvzf\" (UID: \"f85282ae-679b-472b-b395-88f43b596eb0\") " pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:39:59.168218 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.168073    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdgg\" (UniqueName: \"kubernetes.io/projected/891539dc-9b96-4d97-a03a-fa8826801d9b-kube-api-access-rzdgg\") pod \"seaweedfs-79f457656b-znl9n\" (UID: \"891539dc-9b96-4d97-a03a-fa8826801d9b\") " pod="kserve/seaweedfs-79f457656b-znl9n"
May 20 15:39:59.227083 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.227051    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-znl9n"
May 20 15:39:59.360612 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.360554    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-znl9n"]
May 20 15:39:59.362858 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:39:59.362834    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891539dc_9b96_4d97_a03a_fa8826801d9b.slice/crio-5f2a4f7bdcc781d302608d49094b2a0d6e8c2ed3f99cc80011a602d89c5c23b9 WatchSource:0}: Error finding container 5f2a4f7bdcc781d302608d49094b2a0d6e8c2ed3f99cc80011a602d89c5c23b9: Status 404 returned error can't find the container with id 5f2a4f7bdcc781d302608d49094b2a0d6e8c2ed3f99cc80011a602d89c5c23b9
May 20 15:39:59.563441 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.563345    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c200523-580a-4852-9cff-0318b950ca82-cert\") pod \"kserve-controller-manager-87b4b9896-d2s9d\" (UID: \"5c200523-580a-4852-9cff-0318b950ca82\") " pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:39:59.565886 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.565866    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c200523-580a-4852-9cff-0318b950ca82-cert\") pod \"kserve-controller-manager-87b4b9896-d2s9d\" (UID: \"5c200523-580a-4852-9cff-0318b950ca82\") " pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:39:59.664285 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.664235    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f85282ae-679b-472b-b395-88f43b596eb0-cert\") pod \"llmisvc-controller-manager-8677dc7484-bsvzf\" (UID: \"f85282ae-679b-472b-b395-88f43b596eb0\") " pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:39:59.667168 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.667137    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f85282ae-679b-472b-b395-88f43b596eb0-cert\") pod \"llmisvc-controller-manager-8677dc7484-bsvzf\" (UID: \"f85282ae-679b-472b-b395-88f43b596eb0\") " pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:39:59.768300 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.768259    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:39:59.805298 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.805256    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:39:59.971882 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:39:59.971839    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-znl9n" event={"ID":"891539dc-9b96-4d97-a03a-fa8826801d9b","Type":"ContainerStarted","Data":"5f2a4f7bdcc781d302608d49094b2a0d6e8c2ed3f99cc80011a602d89c5c23b9"}
May 20 15:40:00.001384 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:00.001353    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-8677dc7484-bsvzf"]
May 20 15:40:00.004409 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:40:00.004380    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf85282ae_679b_472b_b395_88f43b596eb0.slice/crio-c175825487f39fb3a780c53510a0398d8628f1e96af4131bb2e895715b018640 WatchSource:0}: Error finding container c175825487f39fb3a780c53510a0398d8628f1e96af4131bb2e895715b018640: Status 404 returned error can't find the container with id c175825487f39fb3a780c53510a0398d8628f1e96af4131bb2e895715b018640
May 20 15:40:00.019059 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:00.019019    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-d2s9d"]
May 20 15:40:00.021629 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:40:00.021595    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c200523_580a_4852_9cff_0318b950ca82.slice/crio-0a3e6c7d6e24f3300513bdc45062b750f7e4f0aa4e8bd2c40657b61dc27d2f4a WatchSource:0}: Error finding container 0a3e6c7d6e24f3300513bdc45062b750f7e4f0aa4e8bd2c40657b61dc27d2f4a: Status 404 returned error can't find the container with id 0a3e6c7d6e24f3300513bdc45062b750f7e4f0aa4e8bd2c40657b61dc27d2f4a
May 20 15:40:00.978393 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:00.978349    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf" event={"ID":"f85282ae-679b-472b-b395-88f43b596eb0","Type":"ContainerStarted","Data":"c175825487f39fb3a780c53510a0398d8628f1e96af4131bb2e895715b018640"}
May 20 15:40:00.981109 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:00.981061    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d" event={"ID":"5c200523-580a-4852-9cff-0318b950ca82","Type":"ContainerStarted","Data":"0a3e6c7d6e24f3300513bdc45062b750f7e4f0aa4e8bd2c40657b61dc27d2f4a"}
May 20 15:40:04.997875 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:04.997837    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf" event={"ID":"f85282ae-679b-472b-b395-88f43b596eb0","Type":"ContainerStarted","Data":"c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f"}
May 20 15:40:04.998345 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:04.997957    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:40:04.999205 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:04.999176    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d" event={"ID":"5c200523-580a-4852-9cff-0318b950ca82","Type":"ContainerStarted","Data":"c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35"}
May 20 15:40:04.999347 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:04.999263    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:40:05.000443 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:05.000421    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-znl9n" event={"ID":"891539dc-9b96-4d97-a03a-fa8826801d9b","Type":"ContainerStarted","Data":"79a07f52885caa5dc1307c32bfae25084d3f2f1c90b9dfd7be2dd746f2f47491"}
May 20 15:40:05.000556 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:05.000539    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-znl9n"
May 20 15:40:05.018883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:05.018831    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf" podStartSLOduration=2.669079232 podStartE2EDuration="7.018817907s" podCreationTimestamp="2026-05-20 15:39:58 +0000 UTC" firstStartedPulling="2026-05-20 15:40:00.006564204 +0000 UTC m=+489.710657773" lastFinishedPulling="2026-05-20 15:40:04.356302861 +0000 UTC m=+494.060396448" observedRunningTime="2026-05-20 15:40:05.017509275 +0000 UTC m=+494.721602868" watchObservedRunningTime="2026-05-20 15:40:05.018817907 +0000 UTC m=+494.722911497"
May 20 15:40:05.038192 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:05.038137    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-znl9n" podStartSLOduration=1.9935524230000001 podStartE2EDuration="7.038117328s" podCreationTimestamp="2026-05-20 15:39:58 +0000 UTC" firstStartedPulling="2026-05-20 15:39:59.364201494 +0000 UTC m=+489.068295062" lastFinishedPulling="2026-05-20 15:40:04.40876639 +0000 UTC m=+494.112859967" observedRunningTime="2026-05-20 15:40:05.036467217 +0000 UTC m=+494.740560809" watchObservedRunningTime="2026-05-20 15:40:05.038117328 +0000 UTC m=+494.742210918"
May 20 15:40:05.053938 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:05.053889    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d" podStartSLOduration=2.801521269 podStartE2EDuration="7.053874444s" podCreationTimestamp="2026-05-20 15:39:58 +0000 UTC" firstStartedPulling="2026-05-20 15:40:00.023469155 +0000 UTC m=+489.727562723" lastFinishedPulling="2026-05-20 15:40:04.275822327 +0000 UTC m=+493.979915898" observedRunningTime="2026-05-20 15:40:05.052552793 +0000 UTC m=+494.756646387" watchObservedRunningTime="2026-05-20 15:40:05.053874444 +0000 UTC m=+494.757968080"
May 20 15:40:11.006340 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:11.006310    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-znl9n"
May 20 15:40:36.006556 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:36.006472    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:40:36.009183 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:36.009162    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:40:37.324603 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.324555    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"]
May 20 15:40:37.328220 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.328197    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:37.331154 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.331127    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 15:40:37.331291 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.331156    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 15:40:37.332294 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.332273    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g25pr\""
May 20 15:40:37.338272 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.338247    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"]
May 20 15:40:37.411302 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.411257    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgrb\" (UniqueName: \"kubernetes.io/projected/af5a5e45-c005-4478-88f1-5b3ec46013df-kube-api-access-lkgrb\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:37.411482 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.411362    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:37.411482 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.411402    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:37.512897 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.512858    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:37.513027 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.512915    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:37.513027 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.512959    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgrb\" (UniqueName: \"kubernetes.io/projected/af5a5e45-c005-4478-88f1-5b3ec46013df-kube-api-access-lkgrb\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:37.513330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.513303    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:37.513369 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.513311    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:37.524447 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.524418    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgrb\" (UniqueName: \"kubernetes.io/projected/af5a5e45-c005-4478-88f1-5b3ec46013df-kube-api-access-lkgrb\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:37.638095 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.638051    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:37.770172 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:37.770146    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"]
May 20 15:40:37.774259 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:40:37.774225    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5a5e45_c005_4478_88f1_5b3ec46013df.slice/crio-abf5fd67d1f48680e8384b798a7186b0b17d22523732875edc741a19f000c30d WatchSource:0}: Error finding container abf5fd67d1f48680e8384b798a7186b0b17d22523732875edc741a19f000c30d: Status 404 returned error can't find the container with id abf5fd67d1f48680e8384b798a7186b0b17d22523732875edc741a19f000c30d
May 20 15:40:38.116526 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:38.116483    2567 generic.go:358] "Generic (PLEG): container finished" podID="af5a5e45-c005-4478-88f1-5b3ec46013df" containerID="52aca4565312f3e98d970ab277213863f2fc00d9840c086bef636be4bb4cc74f" exitCode=0
May 20 15:40:38.116722 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:38.116552    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6" event={"ID":"af5a5e45-c005-4478-88f1-5b3ec46013df","Type":"ContainerDied","Data":"52aca4565312f3e98d970ab277213863f2fc00d9840c086bef636be4bb4cc74f"}
May 20 15:40:38.116722 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:38.116618    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6" event={"ID":"af5a5e45-c005-4478-88f1-5b3ec46013df","Type":"ContainerStarted","Data":"abf5fd67d1f48680e8384b798a7186b0b17d22523732875edc741a19f000c30d"}
May 20 15:40:41.131784 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:41.131749    2567 generic.go:358] "Generic (PLEG): container finished" podID="af5a5e45-c005-4478-88f1-5b3ec46013df" containerID="c076b34f33422d15eaccf75ac459434da3118fdefa67e0a5ff4b95196b91b5ab" exitCode=0
May 20 15:40:41.132171 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:41.131827    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6" event={"ID":"af5a5e45-c005-4478-88f1-5b3ec46013df","Type":"ContainerDied","Data":"c076b34f33422d15eaccf75ac459434da3118fdefa67e0a5ff4b95196b91b5ab"}
May 20 15:40:42.137330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:42.137294    2567 generic.go:358] "Generic (PLEG): container finished" podID="af5a5e45-c005-4478-88f1-5b3ec46013df" containerID="3528f35d57a2a6946024d2287e131d2135fa84c94c735d18830187d91ccc65d6" exitCode=0
May 20 15:40:42.137819 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:42.137395    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6" event={"ID":"af5a5e45-c005-4478-88f1-5b3ec46013df","Type":"ContainerDied","Data":"3528f35d57a2a6946024d2287e131d2135fa84c94c735d18830187d91ccc65d6"}
May 20 15:40:43.272712 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:43.272679    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:43.370729 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:43.370693    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkgrb\" (UniqueName: \"kubernetes.io/projected/af5a5e45-c005-4478-88f1-5b3ec46013df-kube-api-access-lkgrb\") pod \"af5a5e45-c005-4478-88f1-5b3ec46013df\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") "
May 20 15:40:43.370925 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:43.370802    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-bundle\") pod \"af5a5e45-c005-4478-88f1-5b3ec46013df\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") "
May 20 15:40:43.370925 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:43.370839    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-util\") pod \"af5a5e45-c005-4478-88f1-5b3ec46013df\" (UID: \"af5a5e45-c005-4478-88f1-5b3ec46013df\") "
May 20 15:40:43.371488 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:43.371458    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-bundle" (OuterVolumeSpecName: "bundle") pod "af5a5e45-c005-4478-88f1-5b3ec46013df" (UID: "af5a5e45-c005-4478-88f1-5b3ec46013df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:40:43.373025 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:43.373002    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5a5e45-c005-4478-88f1-5b3ec46013df-kube-api-access-lkgrb" (OuterVolumeSpecName: "kube-api-access-lkgrb") pod "af5a5e45-c005-4478-88f1-5b3ec46013df" (UID: "af5a5e45-c005-4478-88f1-5b3ec46013df"). InnerVolumeSpecName "kube-api-access-lkgrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:40:43.375817 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:43.375793    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-util" (OuterVolumeSpecName: "util") pod "af5a5e45-c005-4478-88f1-5b3ec46013df" (UID: "af5a5e45-c005-4478-88f1-5b3ec46013df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:40:43.471673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:43.471565    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkgrb\" (UniqueName: \"kubernetes.io/projected/af5a5e45-c005-4478-88f1-5b3ec46013df-kube-api-access-lkgrb\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:40:43.471673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:43.471621    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:40:43.471673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:43.471636    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af5a5e45-c005-4478-88f1-5b3ec46013df-util\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:40:44.146231 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:44.146192    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6" event={"ID":"af5a5e45-c005-4478-88f1-5b3ec46013df","Type":"ContainerDied","Data":"abf5fd67d1f48680e8384b798a7186b0b17d22523732875edc741a19f000c30d"}
May 20 15:40:44.146231 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:44.146228    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf5fd67d1f48680e8384b798a7186b0b17d22523732875edc741a19f000c30d"
May 20 15:40:44.146486 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:44.146283    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19dc4qj6"
May 20 15:40:50.037998 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.037960    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7"]
May 20 15:40:50.038372 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.038350    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5a5e45-c005-4478-88f1-5b3ec46013df" containerName="util"
May 20 15:40:50.038372 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.038361    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5a5e45-c005-4478-88f1-5b3ec46013df" containerName="util"
May 20 15:40:50.038372 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.038371    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5a5e45-c005-4478-88f1-5b3ec46013df" containerName="pull"
May 20 15:40:50.038467 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.038376    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5a5e45-c005-4478-88f1-5b3ec46013df" containerName="pull"
May 20 15:40:50.038467 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.038384    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5a5e45-c005-4478-88f1-5b3ec46013df" containerName="extract"
May 20 15:40:50.038467 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.038389    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5a5e45-c005-4478-88f1-5b3ec46013df" containerName="extract"
May 20 15:40:50.038467 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.038450    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="af5a5e45-c005-4478-88f1-5b3ec46013df" containerName="extract"
May 20 15:40:50.043307 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.043287    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7"
May 20 15:40:50.049102 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.049074    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
May 20 15:40:50.050008 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.049674    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
May 20 15:40:50.051701 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.051682    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7nqcj\""
May 20 15:40:50.064373 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.064338    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7"]
May 20 15:40:50.137027 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.136982    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5xq\" (UniqueName: \"kubernetes.io/projected/512516e0-f493-420a-afdd-52b80796bc50-kube-api-access-tt5xq\") pod \"cert-manager-operator-controller-manager-54b9655956-b4pg7\" (UID: \"512516e0-f493-420a-afdd-52b80796bc50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7"
May 20 15:40:50.137187 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.137064    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/512516e0-f493-420a-afdd-52b80796bc50-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-b4pg7\" (UID: \"512516e0-f493-420a-afdd-52b80796bc50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7"
May 20 15:40:50.238367 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.238327    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5xq\" (UniqueName: \"kubernetes.io/projected/512516e0-f493-420a-afdd-52b80796bc50-kube-api-access-tt5xq\") pod \"cert-manager-operator-controller-manager-54b9655956-b4pg7\" (UID: \"512516e0-f493-420a-afdd-52b80796bc50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7"
May 20 15:40:50.238521 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.238392    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/512516e0-f493-420a-afdd-52b80796bc50-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-b4pg7\" (UID: \"512516e0-f493-420a-afdd-52b80796bc50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7"
May 20 15:40:50.238770 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.238753    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/512516e0-f493-420a-afdd-52b80796bc50-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-b4pg7\" (UID: \"512516e0-f493-420a-afdd-52b80796bc50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7"
May 20 15:40:50.247895 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.247875    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5xq\" (UniqueName: \"kubernetes.io/projected/512516e0-f493-420a-afdd-52b80796bc50-kube-api-access-tt5xq\") pod \"cert-manager-operator-controller-manager-54b9655956-b4pg7\" (UID: \"512516e0-f493-420a-afdd-52b80796bc50\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7"
May 20 15:40:50.352198 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.352166    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7"
May 20 15:40:50.496063 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:50.496035    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7"]
May 20 15:40:50.498659 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:40:50.498626    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512516e0_f493_420a_afdd_52b80796bc50.slice/crio-a4c10c84184d726a5779381ff0e1e8661af10ef71253ae4a1ca34e1a321a3bfc WatchSource:0}: Error finding container a4c10c84184d726a5779381ff0e1e8661af10ef71253ae4a1ca34e1a321a3bfc: Status 404 returned error can't find the container with id a4c10c84184d726a5779381ff0e1e8661af10ef71253ae4a1ca34e1a321a3bfc
May 20 15:40:51.169730 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:51.169692    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7" event={"ID":"512516e0-f493-420a-afdd-52b80796bc50","Type":"ContainerStarted","Data":"a4c10c84184d726a5779381ff0e1e8661af10ef71253ae4a1ca34e1a321a3bfc"}
May 20 15:40:55.188459 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:55.188416    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7" event={"ID":"512516e0-f493-420a-afdd-52b80796bc50","Type":"ContainerStarted","Data":"8a37c0e57134b513835b2ef46816b64096c3d2b7cdb17f750590fe948f873ad4"}
May 20 15:40:55.213472 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:55.213406    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-b4pg7" podStartSLOduration=1.101958037 podStartE2EDuration="5.213382947s" podCreationTimestamp="2026-05-20 15:40:50 +0000 UTC" firstStartedPulling="2026-05-20 15:40:50.501054831 +0000 UTC m=+540.205148398" lastFinishedPulling="2026-05-20 15:40:54.612479739 +0000 UTC m=+544.316573308" observedRunningTime="2026-05-20 15:40:55.2098288 +0000 UTC m=+544.913922393" watchObservedRunningTime="2026-05-20 15:40:55.213382947 +0000 UTC m=+544.917476537"
May 20 15:40:59.453232 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.453197    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"]
May 20 15:40:59.457055 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.457037    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:40:59.459904 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.459879    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 15:40:59.461014 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.460994    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g25pr\""
May 20 15:40:59.461014 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.461010    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 15:40:59.465438 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.465413    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"]
May 20 15:40:59.510023 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.509979    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg46z\" (UniqueName: \"kubernetes.io/projected/7286adcd-36ed-4c79-a10f-b90f636cd747-kube-api-access-kg46z\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:40:59.510231 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.510034    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:40:59.510231 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.510129    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:40:59.611525 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.611486    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:40:59.611780 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.611573    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kg46z\" (UniqueName: \"kubernetes.io/projected/7286adcd-36ed-4c79-a10f-b90f636cd747-kube-api-access-kg46z\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:40:59.611780 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.611623    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:40:59.611910 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.611895    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:40:59.611961 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.611921    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:40:59.621134 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.621096    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg46z\" (UniqueName: \"kubernetes.io/projected/7286adcd-36ed-4c79-a10f-b90f636cd747-kube-api-access-kg46z\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:40:59.767647 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.767550    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:40:59.910857 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:40:59.910833    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"]
May 20 15:40:59.913436 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:40:59.913405    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7286adcd_36ed_4c79_a10f_b90f636cd747.slice/crio-fc6487ceb2b14187211433bd958988bb6d94c7201ae37c21027df76589f03ef9 WatchSource:0}: Error finding container fc6487ceb2b14187211433bd958988bb6d94c7201ae37c21027df76589f03ef9: Status 404 returned error can't find the container with id fc6487ceb2b14187211433bd958988bb6d94c7201ae37c21027df76589f03ef9
May 20 15:41:00.208407 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:00.208374    2567 generic.go:358] "Generic (PLEG): container finished" podID="7286adcd-36ed-4c79-a10f-b90f636cd747" containerID="dd4e905786d5fb320000f761dce06990f10db80e3db43f148c0afafad110fc9b" exitCode=0
May 20 15:41:00.208612 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:00.208424    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d" event={"ID":"7286adcd-36ed-4c79-a10f-b90f636cd747","Type":"ContainerDied","Data":"dd4e905786d5fb320000f761dce06990f10db80e3db43f148c0afafad110fc9b"}
May 20 15:41:00.208612 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:00.208448    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d" event={"ID":"7286adcd-36ed-4c79-a10f-b90f636cd747","Type":"ContainerStarted","Data":"fc6487ceb2b14187211433bd958988bb6d94c7201ae37c21027df76589f03ef9"}
May 20 15:41:02.218035 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:02.218000    2567 generic.go:358] "Generic (PLEG): container finished" podID="7286adcd-36ed-4c79-a10f-b90f636cd747" containerID="4ed5eb74649c160247e3067633a95c3c00160bdf04ee8e6ef0024cc089984d45" exitCode=0
May 20 15:41:02.218401 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:02.218047    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d" event={"ID":"7286adcd-36ed-4c79-a10f-b90f636cd747","Type":"ContainerDied","Data":"4ed5eb74649c160247e3067633a95c3c00160bdf04ee8e6ef0024cc089984d45"}
May 20 15:41:03.223761 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:03.223726    2567 generic.go:358] "Generic (PLEG): container finished" podID="7286adcd-36ed-4c79-a10f-b90f636cd747" containerID="af07b4038924a73c8a84bf1d49d6d720d9808f529a35346eb800b9335c37f547" exitCode=0
May 20 15:41:03.224156 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:03.223816    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d" event={"ID":"7286adcd-36ed-4c79-a10f-b90f636cd747","Type":"ContainerDied","Data":"af07b4038924a73c8a84bf1d49d6d720d9808f529a35346eb800b9335c37f547"}
May 20 15:41:04.358403 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:04.358375    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:41:04.456562 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:04.456521    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg46z\" (UniqueName: \"kubernetes.io/projected/7286adcd-36ed-4c79-a10f-b90f636cd747-kube-api-access-kg46z\") pod \"7286adcd-36ed-4c79-a10f-b90f636cd747\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") "
May 20 15:41:04.456781 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:04.456576    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-bundle\") pod \"7286adcd-36ed-4c79-a10f-b90f636cd747\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") "
May 20 15:41:04.456781 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:04.456647    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-util\") pod \"7286adcd-36ed-4c79-a10f-b90f636cd747\" (UID: \"7286adcd-36ed-4c79-a10f-b90f636cd747\") "
May 20 15:41:04.457064 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:04.457036    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-bundle" (OuterVolumeSpecName: "bundle") pod "7286adcd-36ed-4c79-a10f-b90f636cd747" (UID: "7286adcd-36ed-4c79-a10f-b90f636cd747"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:41:04.458949 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:04.458919    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7286adcd-36ed-4c79-a10f-b90f636cd747-kube-api-access-kg46z" (OuterVolumeSpecName: "kube-api-access-kg46z") pod "7286adcd-36ed-4c79-a10f-b90f636cd747" (UID: "7286adcd-36ed-4c79-a10f-b90f636cd747"). InnerVolumeSpecName "kube-api-access-kg46z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:41:04.462156 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:04.462132    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-util" (OuterVolumeSpecName: "util") pod "7286adcd-36ed-4c79-a10f-b90f636cd747" (UID: "7286adcd-36ed-4c79-a10f-b90f636cd747"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:41:04.557764 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:04.557663    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg46z\" (UniqueName: \"kubernetes.io/projected/7286adcd-36ed-4c79-a10f-b90f636cd747-kube-api-access-kg46z\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:41:04.557764 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:04.557708    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:41:04.557764 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:04.557718    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7286adcd-36ed-4c79-a10f-b90f636cd747-util\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:41:05.232121 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:05.232086    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d" event={"ID":"7286adcd-36ed-4c79-a10f-b90f636cd747","Type":"ContainerDied","Data":"fc6487ceb2b14187211433bd958988bb6d94c7201ae37c21027df76589f03ef9"}
May 20 15:41:05.232121 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:05.232120    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6487ceb2b14187211433bd958988bb6d94c7201ae37c21027df76589f03ef9"
May 20 15:41:05.232327 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:05.232140    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4hz4d"
May 20 15:41:27.695780 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.695743    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"]
May 20 15:41:27.696246 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.696112    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7286adcd-36ed-4c79-a10f-b90f636cd747" containerName="pull"
May 20 15:41:27.696246 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.696123    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7286adcd-36ed-4c79-a10f-b90f636cd747" containerName="pull"
May 20 15:41:27.696246 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.696133    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7286adcd-36ed-4c79-a10f-b90f636cd747" containerName="extract"
May 20 15:41:27.696246 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.696138    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7286adcd-36ed-4c79-a10f-b90f636cd747" containerName="extract"
May 20 15:41:27.696246 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.696147    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7286adcd-36ed-4c79-a10f-b90f636cd747" containerName="util"
May 20 15:41:27.696246 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.696152    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7286adcd-36ed-4c79-a10f-b90f636cd747" containerName="util"
May 20 15:41:27.696246 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.696205    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="7286adcd-36ed-4c79-a10f-b90f636cd747" containerName="extract"
May 20 15:41:27.704666 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.704643    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:27.708727 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.708522    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 15:41:27.708727 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.708544    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 15:41:27.708727 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.708612    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g25pr\""
May 20 15:41:27.709780 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.709755    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"]
May 20 15:41:27.877329 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.877287    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:27.877515 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.877348    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7jbc\" (UniqueName: \"kubernetes.io/projected/ce1e7cba-5c3a-4785-a545-1c290379abad-kube-api-access-m7jbc\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:27.877515 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.877428    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:27.978301 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.978215    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m7jbc\" (UniqueName: \"kubernetes.io/projected/ce1e7cba-5c3a-4785-a545-1c290379abad-kube-api-access-m7jbc\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:27.978301 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.978274    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:27.978498 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.978337    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:27.978689 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.978669    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:27.978736 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.978710    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:27.987627 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:27.987600    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7jbc\" (UniqueName: \"kubernetes.io/projected/ce1e7cba-5c3a-4785-a545-1c290379abad-kube-api-access-m7jbc\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:28.015573 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:28.015532    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:28.149614 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:28.149569    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"]
May 20 15:41:28.151475 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:41:28.151447    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1e7cba_5c3a_4785_a545_1c290379abad.slice/crio-921dcbcdaa3804c2bd952ea6abd72f599bfc2a41e1874822cb3c27a1585b765b WatchSource:0}: Error finding container 921dcbcdaa3804c2bd952ea6abd72f599bfc2a41e1874822cb3c27a1585b765b: Status 404 returned error can't find the container with id 921dcbcdaa3804c2bd952ea6abd72f599bfc2a41e1874822cb3c27a1585b765b
May 20 15:41:28.313413 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:28.313376    2567 generic.go:358] "Generic (PLEG): container finished" podID="ce1e7cba-5c3a-4785-a545-1c290379abad" containerID="0e5a4249dc860eb394bc2beddd46ac74510723b4676566c27b05f3dc2adba259" exitCode=0
May 20 15:41:28.313571 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:28.313453    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4" event={"ID":"ce1e7cba-5c3a-4785-a545-1c290379abad","Type":"ContainerDied","Data":"0e5a4249dc860eb394bc2beddd46ac74510723b4676566c27b05f3dc2adba259"}
May 20 15:41:28.313571 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:28.313484    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4" event={"ID":"ce1e7cba-5c3a-4785-a545-1c290379abad","Type":"ContainerStarted","Data":"921dcbcdaa3804c2bd952ea6abd72f599bfc2a41e1874822cb3c27a1585b765b"}
May 20 15:41:29.318643 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:29.318601    2567 generic.go:358] "Generic (PLEG): container finished" podID="ce1e7cba-5c3a-4785-a545-1c290379abad" containerID="4d76e0b364e2e68f04a28eb1adb0bbc0675f415db22bc57dec51620fb879d237" exitCode=0
May 20 15:41:29.319138 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:29.318685    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4" event={"ID":"ce1e7cba-5c3a-4785-a545-1c290379abad","Type":"ContainerDied","Data":"4d76e0b364e2e68f04a28eb1adb0bbc0675f415db22bc57dec51620fb879d237"}
May 20 15:41:30.323480 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:30.323441    2567 generic.go:358] "Generic (PLEG): container finished" podID="ce1e7cba-5c3a-4785-a545-1c290379abad" containerID="1fec1f1ffdf627f02b3db050c8bd9581742efa1cf21c4f9dd17dd718687bc9fd" exitCode=0
May 20 15:41:30.323978 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:30.323512    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4" event={"ID":"ce1e7cba-5c3a-4785-a545-1c290379abad","Type":"ContainerDied","Data":"1fec1f1ffdf627f02b3db050c8bd9581742efa1cf21c4f9dd17dd718687bc9fd"}
May 20 15:41:31.468399 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:31.468371    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:31.609327 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:31.609292    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-bundle\") pod \"ce1e7cba-5c3a-4785-a545-1c290379abad\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") "
May 20 15:41:31.609525 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:31.609361    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7jbc\" (UniqueName: \"kubernetes.io/projected/ce1e7cba-5c3a-4785-a545-1c290379abad-kube-api-access-m7jbc\") pod \"ce1e7cba-5c3a-4785-a545-1c290379abad\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") "
May 20 15:41:31.609525 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:31.609444    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-util\") pod \"ce1e7cba-5c3a-4785-a545-1c290379abad\" (UID: \"ce1e7cba-5c3a-4785-a545-1c290379abad\") "
May 20 15:41:31.610319 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:31.610283    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-bundle" (OuterVolumeSpecName: "bundle") pod "ce1e7cba-5c3a-4785-a545-1c290379abad" (UID: "ce1e7cba-5c3a-4785-a545-1c290379abad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:41:31.611572 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:31.611552    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1e7cba-5c3a-4785-a545-1c290379abad-kube-api-access-m7jbc" (OuterVolumeSpecName: "kube-api-access-m7jbc") pod "ce1e7cba-5c3a-4785-a545-1c290379abad" (UID: "ce1e7cba-5c3a-4785-a545-1c290379abad"). InnerVolumeSpecName "kube-api-access-m7jbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:41:31.614836 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:31.614803    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-util" (OuterVolumeSpecName: "util") pod "ce1e7cba-5c3a-4785-a545-1c290379abad" (UID: "ce1e7cba-5c3a-4785-a545-1c290379abad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:41:31.710507 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:31.710467    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:41:31.710507 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:31.710502    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7jbc\" (UniqueName: \"kubernetes.io/projected/ce1e7cba-5c3a-4785-a545-1c290379abad-kube-api-access-m7jbc\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:41:31.710507 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:31.710518    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce1e7cba-5c3a-4785-a545-1c290379abad-util\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:41:32.332033 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:32.332003    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4"
May 20 15:41:32.332207 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:32.332016    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835lgxc4" event={"ID":"ce1e7cba-5c3a-4785-a545-1c290379abad","Type":"ContainerDied","Data":"921dcbcdaa3804c2bd952ea6abd72f599bfc2a41e1874822cb3c27a1585b765b"}
May 20 15:41:32.332207 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:32.332065    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="921dcbcdaa3804c2bd952ea6abd72f599bfc2a41e1874822cb3c27a1585b765b"
May 20 15:41:37.722635 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.722600    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"]
May 20 15:41:37.723081 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.722954    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1e7cba-5c3a-4785-a545-1c290379abad" containerName="extract"
May 20 15:41:37.723081 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.722965    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1e7cba-5c3a-4785-a545-1c290379abad" containerName="extract"
May 20 15:41:37.723081 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.722977    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1e7cba-5c3a-4785-a545-1c290379abad" containerName="util"
May 20 15:41:37.723081 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.722983    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1e7cba-5c3a-4785-a545-1c290379abad" containerName="util"
May 20 15:41:37.723081 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.722989    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1e7cba-5c3a-4785-a545-1c290379abad" containerName="pull"
May 20 15:41:37.723081 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.722994    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1e7cba-5c3a-4785-a545-1c290379abad" containerName="pull"
May 20 15:41:37.723081 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.723061    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="ce1e7cba-5c3a-4785-a545-1c290379abad" containerName="extract"
May 20 15:41:37.727212 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.727188    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:37.730007 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.729982    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 15:41:37.730142 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.730014    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g25pr\""
May 20 15:41:37.730142 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.729982    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 15:41:37.737404 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.737378    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"]
May 20 15:41:37.753880 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.753847    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxj29\" (UniqueName: \"kubernetes.io/projected/58364226-c767-4f88-8924-be41cc7d2e9d-kube-api-access-zxj29\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:37.754052 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.753887    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:37.754052 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.753979    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:37.855293 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.855253    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:37.855471 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.855312    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:37.855471 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.855375    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zxj29\" (UniqueName: \"kubernetes.io/projected/58364226-c767-4f88-8924-be41cc7d2e9d-kube-api-access-zxj29\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:37.855798 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.855774    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:37.855848 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.855783    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:37.873829 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:37.873791    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxj29\" (UniqueName: \"kubernetes.io/projected/58364226-c767-4f88-8924-be41cc7d2e9d-kube-api-access-zxj29\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:38.037414 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:38.037321    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:38.189877 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:41:38.189840    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58364226_c767_4f88_8924_be41cc7d2e9d.slice/crio-571dee7e6f07c295152f1fc17c774911bc5dff40410263dbfdc6a272b6486cd6 WatchSource:0}: Error finding container 571dee7e6f07c295152f1fc17c774911bc5dff40410263dbfdc6a272b6486cd6: Status 404 returned error can't find the container with id 571dee7e6f07c295152f1fc17c774911bc5dff40410263dbfdc6a272b6486cd6
May 20 15:41:38.193205 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:38.193182    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"]
May 20 15:41:38.355673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:38.355636    2567 generic.go:358] "Generic (PLEG): container finished" podID="58364226-c767-4f88-8924-be41cc7d2e9d" containerID="a4f51d08b73f0155833e7212196d383ec30cc3e7ae15b97d62e5e8591f895324" exitCode=0
May 20 15:41:38.355909 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:38.355731    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7" event={"ID":"58364226-c767-4f88-8924-be41cc7d2e9d","Type":"ContainerDied","Data":"a4f51d08b73f0155833e7212196d383ec30cc3e7ae15b97d62e5e8591f895324"}
May 20 15:41:38.355909 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:38.355776    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7" event={"ID":"58364226-c767-4f88-8924-be41cc7d2e9d","Type":"ContainerStarted","Data":"571dee7e6f07c295152f1fc17c774911bc5dff40410263dbfdc6a272b6486cd6"}
May 20 15:41:39.024058 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.024019    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"]
May 20 15:41:39.026512 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.026495    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"
May 20 15:41:39.032562 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.032534    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"openshift-service-ca.crt\""
May 20 15:41:39.032755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.032534    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators\"/\"servicemesh-operator3-dockercfg-fdjww\""
May 20 15:41:39.032755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.032566    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"kube-root-ca.crt\""
May 20 15:41:39.040391 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.040360    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"]
May 20 15:41:39.066923 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.066883    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5cgz\" (UniqueName: \"kubernetes.io/projected/8b9b7f0b-115a-49b9-bee9-4bed7bd1d819-kube-api-access-m5cgz\") pod \"servicemesh-operator3-55f49c5f94-qk6vs\" (UID: \"8b9b7f0b-115a-49b9-bee9-4bed7bd1d819\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"
May 20 15:41:39.067094 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.066927    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/8b9b7f0b-115a-49b9-bee9-4bed7bd1d819-operator-config\") pod \"servicemesh-operator3-55f49c5f94-qk6vs\" (UID: \"8b9b7f0b-115a-49b9-bee9-4bed7bd1d819\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"
May 20 15:41:39.168238 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.168156    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m5cgz\" (UniqueName: \"kubernetes.io/projected/8b9b7f0b-115a-49b9-bee9-4bed7bd1d819-kube-api-access-m5cgz\") pod \"servicemesh-operator3-55f49c5f94-qk6vs\" (UID: \"8b9b7f0b-115a-49b9-bee9-4bed7bd1d819\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"
May 20 15:41:39.168238 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.168204    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/8b9b7f0b-115a-49b9-bee9-4bed7bd1d819-operator-config\") pod \"servicemesh-operator3-55f49c5f94-qk6vs\" (UID: \"8b9b7f0b-115a-49b9-bee9-4bed7bd1d819\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"
May 20 15:41:39.170963 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.170941    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/8b9b7f0b-115a-49b9-bee9-4bed7bd1d819-operator-config\") pod \"servicemesh-operator3-55f49c5f94-qk6vs\" (UID: \"8b9b7f0b-115a-49b9-bee9-4bed7bd1d819\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"
May 20 15:41:39.186068 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.186037    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5cgz\" (UniqueName: \"kubernetes.io/projected/8b9b7f0b-115a-49b9-bee9-4bed7bd1d819-kube-api-access-m5cgz\") pod \"servicemesh-operator3-55f49c5f94-qk6vs\" (UID: \"8b9b7f0b-115a-49b9-bee9-4bed7bd1d819\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"
May 20 15:41:39.336673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.336640    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"
May 20 15:41:39.362264 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.362210    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7" event={"ID":"58364226-c767-4f88-8924-be41cc7d2e9d","Type":"ContainerStarted","Data":"1ea268e4ec773b98d7f73129500baa94dd969a43e30e2e7f83e1db949bba0ca1"}
May 20 15:41:39.486397 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:39.486330    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"]
May 20 15:41:39.488738 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:41:39.488711    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9b7f0b_115a_49b9_bee9_4bed7bd1d819.slice/crio-ebe097c6933d863f7c95e4a0d22b7ccb842eb75690b59452e7d60d18033710f9 WatchSource:0}: Error finding container ebe097c6933d863f7c95e4a0d22b7ccb842eb75690b59452e7d60d18033710f9: Status 404 returned error can't find the container with id ebe097c6933d863f7c95e4a0d22b7ccb842eb75690b59452e7d60d18033710f9
May 20 15:41:40.367086 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:40.367048    2567 generic.go:358] "Generic (PLEG): container finished" podID="58364226-c767-4f88-8924-be41cc7d2e9d" containerID="1ea268e4ec773b98d7f73129500baa94dd969a43e30e2e7f83e1db949bba0ca1" exitCode=0
May 20 15:41:40.367563 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:40.367129    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7" event={"ID":"58364226-c767-4f88-8924-be41cc7d2e9d","Type":"ContainerDied","Data":"1ea268e4ec773b98d7f73129500baa94dd969a43e30e2e7f83e1db949bba0ca1"}
May 20 15:41:40.368492 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:40.368468    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs" event={"ID":"8b9b7f0b-115a-49b9-bee9-4bed7bd1d819","Type":"ContainerStarted","Data":"ebe097c6933d863f7c95e4a0d22b7ccb842eb75690b59452e7d60d18033710f9"}
May 20 15:41:41.375215 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:41.375171    2567 generic.go:358] "Generic (PLEG): container finished" podID="58364226-c767-4f88-8924-be41cc7d2e9d" containerID="dde81f82beee54aba9ebb79f82f03ae9ba7ab84f0ae51842310fdf8ad6c04859" exitCode=0
May 20 15:41:41.375667 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:41.375257    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7" event={"ID":"58364226-c767-4f88-8924-be41cc7d2e9d","Type":"ContainerDied","Data":"dde81f82beee54aba9ebb79f82f03ae9ba7ab84f0ae51842310fdf8ad6c04859"}
May 20 15:41:42.864848 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:42.864819    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:42.902792 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:42.902764    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxj29\" (UniqueName: \"kubernetes.io/projected/58364226-c767-4f88-8924-be41cc7d2e9d-kube-api-access-zxj29\") pod \"58364226-c767-4f88-8924-be41cc7d2e9d\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") "
May 20 15:41:42.902903 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:42.902852    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-bundle\") pod \"58364226-c767-4f88-8924-be41cc7d2e9d\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") "
May 20 15:41:42.902903 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:42.902878    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-util\") pod \"58364226-c767-4f88-8924-be41cc7d2e9d\" (UID: \"58364226-c767-4f88-8924-be41cc7d2e9d\") "
May 20 15:41:42.903742 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:42.903714    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-bundle" (OuterVolumeSpecName: "bundle") pod "58364226-c767-4f88-8924-be41cc7d2e9d" (UID: "58364226-c767-4f88-8924-be41cc7d2e9d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:41:42.905004 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:42.904974    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58364226-c767-4f88-8924-be41cc7d2e9d-kube-api-access-zxj29" (OuterVolumeSpecName: "kube-api-access-zxj29") pod "58364226-c767-4f88-8924-be41cc7d2e9d" (UID: "58364226-c767-4f88-8924-be41cc7d2e9d"). InnerVolumeSpecName "kube-api-access-zxj29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:41:42.909558 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:42.909533    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-util" (OuterVolumeSpecName: "util") pod "58364226-c767-4f88-8924-be41cc7d2e9d" (UID: "58364226-c767-4f88-8924-be41cc7d2e9d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:41:43.004271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:43.004231    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxj29\" (UniqueName: \"kubernetes.io/projected/58364226-c767-4f88-8924-be41cc7d2e9d-kube-api-access-zxj29\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:41:43.004271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:43.004260    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:41:43.004271 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:43.004272    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58364226-c767-4f88-8924-be41cc7d2e9d-util\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:41:43.384866 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:43.384827    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs" event={"ID":"8b9b7f0b-115a-49b9-bee9-4bed7bd1d819","Type":"ContainerStarted","Data":"7f9394efdb911712f3b9b6b57973450f7c3d459153e641844c02a9c726acb776"}
May 20 15:41:43.385063 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:43.384954    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"
May 20 15:41:43.386744 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:43.386721    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7" event={"ID":"58364226-c767-4f88-8924-be41cc7d2e9d","Type":"ContainerDied","Data":"571dee7e6f07c295152f1fc17c774911bc5dff40410263dbfdc6a272b6486cd6"}
May 20 15:41:43.386744 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:43.386743    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762azc4j7"
May 20 15:41:43.386884 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:43.386747    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="571dee7e6f07c295152f1fc17c774911bc5dff40410263dbfdc6a272b6486cd6"
May 20 15:41:43.405068 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:43.405022    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs" podStartSLOduration=1.0164416 podStartE2EDuration="4.405006416s" podCreationTimestamp="2026-05-20 15:41:39 +0000 UTC" firstStartedPulling="2026-05-20 15:41:39.491222978 +0000 UTC m=+589.195316546" lastFinishedPulling="2026-05-20 15:41:42.87978778 +0000 UTC m=+592.583881362" observedRunningTime="2026-05-20 15:41:43.404273519 +0000 UTC m=+593.108367110" watchObservedRunningTime="2026-05-20 15:41:43.405006416 +0000 UTC m=+593.109100006"
May 20 15:41:50.999468 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:50.999435    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:41:51.000112 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:51.000086    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:41:54.392371 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:54.392337    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-qk6vs"
May 20 15:41:59.580074 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.580037    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"]
May 20 15:41:59.580616 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.580562    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58364226-c767-4f88-8924-be41cc7d2e9d" containerName="pull"
May 20 15:41:59.580616 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.580600    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="58364226-c767-4f88-8924-be41cc7d2e9d" containerName="pull"
May 20 15:41:59.580616 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.580617    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58364226-c767-4f88-8924-be41cc7d2e9d" containerName="util"
May 20 15:41:59.580774 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.580622    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="58364226-c767-4f88-8924-be41cc7d2e9d" containerName="util"
May 20 15:41:59.580774 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.580638    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58364226-c767-4f88-8924-be41cc7d2e9d" containerName="extract"
May 20 15:41:59.580774 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.580644    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="58364226-c767-4f88-8924-be41cc7d2e9d" containerName="extract"
May 20 15:41:59.580774 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.580732    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="58364226-c767-4f88-8924-be41cc7d2e9d" containerName="extract"
May 20 15:41:59.582877 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.582859    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.585684 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.585663    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
May 20 15:41:59.585953 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.585935    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
May 20 15:41:59.586010 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.585960    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 20 15:41:59.586063 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.586014    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
May 20 15:41:59.586115 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.586073    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 20 15:41:59.586115 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.586099    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j9glp\""
May 20 15:41:59.586346 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.586332    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
May 20 15:41:59.599317 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.599285    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"]
May 20 15:41:59.649533 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.649499    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.649533 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.649537    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.649755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.649570    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6c99\" (UniqueName: \"kubernetes.io/projected/d0fc4c44-95c4-4f18-8d90-8345bab17426-kube-api-access-w6c99\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.649755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.649660    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fc4c44-95c4-4f18-8d90-8345bab17426-local-certs\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.649755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.649689    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-token\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.649755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.649746    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0fc4c44-95c4-4f18-8d90-8345bab17426-cacerts\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.649880 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.649784    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.750719 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.750655    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0fc4c44-95c4-4f18-8d90-8345bab17426-cacerts\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.750904 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.750773    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.750904 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.750827    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.750904 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.750853    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.750904 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.750889    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6c99\" (UniqueName: \"kubernetes.io/projected/d0fc4c44-95c4-4f18-8d90-8345bab17426-kube-api-access-w6c99\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.751090 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.750922    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fc4c44-95c4-4f18-8d90-8345bab17426-local-certs\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.751090 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.750946    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-token\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.751507 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.751479    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.753410 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.753385    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0fc4c44-95c4-4f18-8d90-8345bab17426-local-certs\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.753508 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.753427    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.753655 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.753638    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0fc4c44-95c4-4f18-8d90-8345bab17426-cacerts\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.753765 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.753746    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-kubeconfig\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.759940 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.759919    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0fc4c44-95c4-4f18-8d90-8345bab17426-istio-token\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.760214 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.760194    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6c99\" (UniqueName: \"kubernetes.io/projected/d0fc4c44-95c4-4f18-8d90-8345bab17426-kube-api-access-w6c99\") pod \"istiod-openshift-gateway-798958bb55-pkc4d\" (UID: \"d0fc4c44-95c4-4f18-8d90-8345bab17426\") " pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:41:59.893743 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:41:59.893698    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:42:00.042167 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:00.042138    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"]
May 20 15:42:00.044529 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:42:00.044494    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fc4c44_95c4_4f18_8d90_8345bab17426.slice/crio-3f9b959a4d558790864f9fa3c7de9aaad1ff786b8c0cbed267006f724a49faf3 WatchSource:0}: Error finding container 3f9b959a4d558790864f9fa3c7de9aaad1ff786b8c0cbed267006f724a49faf3: Status 404 returned error can't find the container with id 3f9b959a4d558790864f9fa3c7de9aaad1ff786b8c0cbed267006f724a49faf3
May 20 15:42:00.450637 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:00.450575    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d" event={"ID":"d0fc4c44-95c4-4f18-8d90-8345bab17426","Type":"ContainerStarted","Data":"3f9b959a4d558790864f9fa3c7de9aaad1ff786b8c0cbed267006f724a49faf3"}
May 20 15:42:02.737330 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:02.737282    2567 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
May 20 15:42:02.737602 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:02.737360    2567 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
May 20 15:42:03.466371 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:03.466330    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d" event={"ID":"d0fc4c44-95c4-4f18-8d90-8345bab17426","Type":"ContainerStarted","Data":"45249e92e71a885f3f0a67db126bb4d6c2d2984546031403917893095c96d3e8"}
May 20 15:42:03.466616 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:03.466574    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:42:03.468220 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:03.468192    2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-798958bb55-pkc4d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
May 20 15:42:03.468341 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:03.468246    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d" podUID="d0fc4c44-95c4-4f18-8d90-8345bab17426" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 20 15:42:03.492759 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:03.492705    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d" podStartSLOduration=1.8023811570000001 podStartE2EDuration="4.492691197s" podCreationTimestamp="2026-05-20 15:41:59 +0000 UTC" firstStartedPulling="2026-05-20 15:42:00.046695438 +0000 UTC m=+609.750789010" lastFinishedPulling="2026-05-20 15:42:02.737005476 +0000 UTC m=+612.441099050" observedRunningTime="2026-05-20 15:42:03.491518169 +0000 UTC m=+613.195611764" watchObservedRunningTime="2026-05-20 15:42:03.492691197 +0000 UTC m=+613.196784789"
May 20 15:42:04.470457 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:04.470428    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-798958bb55-pkc4d"
May 20 15:42:05.525436 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.525401    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"]
May 20 15:42:05.528099 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.528075    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.532545 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.532519    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-xwz6z\""
May 20 15:42:05.543495 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.543462    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"]
May 20 15:42:05.603753 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.603721    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-envoy\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.603753 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.603759    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvpb6\" (UniqueName: \"kubernetes.io/projected/82ef32e9-7004-4dc8-bc95-fe1c8413b759-kube-api-access-kvpb6\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.604013 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.603833    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-credential-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.604013 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.603902    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.604013 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.603928    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-workload-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.604013 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.604010    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-token\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.604211 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.604110    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.604211 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.604146    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-workload-certs\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.604312 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.604237    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-data\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.704923 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.704882    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-credential-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705113 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.704937    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705113 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.704972    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-workload-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705113 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.705011    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-token\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705113 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.705062    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705113 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.705083    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-workload-certs\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705312 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.705115    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-data\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705312 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.705149    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-envoy\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705312 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.705182    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kvpb6\" (UniqueName: \"kubernetes.io/projected/82ef32e9-7004-4dc8-bc95-fe1c8413b759-kube-api-access-kvpb6\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705425 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.705383    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-credential-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705425 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.705404    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-workload-socket\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705672 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.705651    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-workload-certs\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.705785 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.705686    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-data\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.706042 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.705789    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.707809 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.707788    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.707905 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.707836    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-envoy\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.714769 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.714743    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ef32e9-7004-4dc8-bc95-fe1c8413b759-istio-token\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.715301 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.715277    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvpb6\" (UniqueName: \"kubernetes.io/projected/82ef32e9-7004-4dc8-bc95-fe1c8413b759-kube-api-access-kvpb6\") pod \"openshift-ai-inference-openshift-default-849cdfbf5f-nm88r\" (UID: \"82ef32e9-7004-4dc8-bc95-fe1c8413b759\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.842810 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.842776    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:05.989949 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:05.989916    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"]
May 20 15:42:05.991842 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:42:05.991812    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ef32e9_7004_4dc8_bc95_fe1c8413b759.slice/crio-041f50dc306ce41711a4d561bab5a68c4cb601b934079017c28e2f42f9502447 WatchSource:0}: Error finding container 041f50dc306ce41711a4d561bab5a68c4cb601b934079017c28e2f42f9502447: Status 404 returned error can't find the container with id 041f50dc306ce41711a4d561bab5a68c4cb601b934079017c28e2f42f9502447
May 20 15:42:06.479307 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:06.479268    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r" event={"ID":"82ef32e9-7004-4dc8-bc95-fe1c8413b759","Type":"ContainerStarted","Data":"041f50dc306ce41711a4d561bab5a68c4cb601b934079017c28e2f42f9502447"}
May 20 15:42:09.051464 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:09.051423    2567 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
May 20 15:42:09.051822 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:09.051526    2567 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
May 20 15:42:09.051822 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:09.051556    2567 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
May 20 15:42:09.493106 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:09.493060    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r" event={"ID":"82ef32e9-7004-4dc8-bc95-fe1c8413b759","Type":"ContainerStarted","Data":"34c97a0a140a0b15c7b31535a807567192cff3657ec6cf48051e4c049f13ea17"}
May 20 15:42:09.520429 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:09.520371    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r" podStartSLOduration=1.463004586 podStartE2EDuration="4.520355302s" podCreationTimestamp="2026-05-20 15:42:05 +0000 UTC" firstStartedPulling="2026-05-20 15:42:05.993839945 +0000 UTC m=+615.697933516" lastFinishedPulling="2026-05-20 15:42:09.051190663 +0000 UTC m=+618.755284232" observedRunningTime="2026-05-20 15:42:09.517475858 +0000 UTC m=+619.221569461" watchObservedRunningTime="2026-05-20 15:42:09.520355302 +0000 UTC m=+619.224448971"
May 20 15:42:09.843380 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:09.843324    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:10.848607 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:10.848556    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:11.500916 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:11.500883    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:11.501840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:11.501822    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-849cdfbf5f-nm88r"
May 20 15:42:14.248069 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.248028    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"]
May 20 15:42:14.252332 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.252305    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:14.255473 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.255453    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 20 15:42:14.255602 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.255456    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 20 15:42:14.256806 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.256786    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-g25pr\""
May 20 15:42:14.260225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.260080    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"]
May 20 15:42:14.349249 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.349213    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"]
May 20 15:42:14.352056 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.352039    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:14.361397 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.361372    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"]
May 20 15:42:14.387677 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.387637    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:14.387848 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.387703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqwb\" (UniqueName: \"kubernetes.io/projected/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-kube-api-access-2tqwb\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:14.387848 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.387779    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:14.454033 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.453998    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"]
May 20 15:42:14.456638 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.456619    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:14.469175 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.469146    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"]
May 20 15:42:14.488519 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.488479    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqwb\" (UniqueName: \"kubernetes.io/projected/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-kube-api-access-2tqwb\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:14.488724 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.488529    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:14.488724 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.488560    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:14.488724 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.488688    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:14.488890 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.488735    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlzm\" (UniqueName: \"kubernetes.io/projected/63b11480-36cf-48d5-8d89-ec3d78ca39a5-kube-api-access-tdlzm\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:14.488890 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.488853    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:14.489006 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.488987    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:14.489116 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.489101    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:14.500567 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.500497    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqwb\" (UniqueName: \"kubernetes.io/projected/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-kube-api-access-2tqwb\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:14.562980 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.562942    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:14.589991 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.589953    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:14.590156 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.590014    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9b7r\" (UniqueName: \"kubernetes.io/projected/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-kube-api-access-k9b7r\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:14.590156 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.590091    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:14.590274 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.590155    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:14.590274 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.590182    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlzm\" (UniqueName: \"kubernetes.io/projected/63b11480-36cf-48d5-8d89-ec3d78ca39a5-kube-api-access-tdlzm\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:14.590274 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.590211    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:14.590492 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.590471    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:14.590492 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.590489    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:14.622785 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.622567    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"]
May 20 15:42:14.627208 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.627183    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:14.630172 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.630040    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlzm\" (UniqueName: \"kubernetes.io/projected/63b11480-36cf-48d5-8d89-ec3d78ca39a5-kube-api-access-tdlzm\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:14.644529 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.644495    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"]
May 20 15:42:14.662084 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.662050    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:14.691262 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.691230    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k9b7r\" (UniqueName: \"kubernetes.io/projected/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-kube-api-access-k9b7r\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:14.691408 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.691299    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:14.691408 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.691359    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:14.691820 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.691733    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:14.691956 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.691909    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:14.701723 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.701691    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9b7r\" (UniqueName: \"kubernetes.io/projected/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-kube-api-access-k9b7r\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:14.733131 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:42:14.733095    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f31b0f9_d7ac_4e86_abe2_010de0a7e76c.slice/crio-c09c76a2923be8e95ca5dfd7abd30ae57da415fb8693158c99c12e83a3fac20b WatchSource:0}: Error finding container c09c76a2923be8e95ca5dfd7abd30ae57da415fb8693158c99c12e83a3fac20b: Status 404 returned error can't find the container with id c09c76a2923be8e95ca5dfd7abd30ae57da415fb8693158c99c12e83a3fac20b
May 20 15:42:14.733490 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.733465    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"]
May 20 15:42:14.765510 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.765479    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:14.792027 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.791987    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:14.792313 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.792291    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:14.792394 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.792379    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd69z\" (UniqueName: \"kubernetes.io/projected/8efd0743-d1ca-4cbd-a724-c18df12d657e-kube-api-access-pd69z\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:14.831600 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.830075    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"]
May 20 15:42:14.893618 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.893555    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:14.893795 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.893645    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:14.893795 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.893729    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pd69z\" (UniqueName: \"kubernetes.io/projected/8efd0743-d1ca-4cbd-a724-c18df12d657e-kube-api-access-pd69z\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:14.894090 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.894061    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:14.894265 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.894076    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:14.902692 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.902670    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd69z\" (UniqueName: \"kubernetes.io/projected/8efd0743-d1ca-4cbd-a724-c18df12d657e-kube-api-access-pd69z\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:14.916122 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.916094    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"]
May 20 15:42:14.921704 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:42:14.921678    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87eba398_ee9a_4623_8ba4_18ee9bf9d9fb.slice/crio-58b7cda2f51e60adeaecc1f2806ff750fa6ae1f7c4a8e3957694f0b8d0d69012 WatchSource:0}: Error finding container 58b7cda2f51e60adeaecc1f2806ff750fa6ae1f7c4a8e3957694f0b8d0d69012: Status 404 returned error can't find the container with id 58b7cda2f51e60adeaecc1f2806ff750fa6ae1f7c4a8e3957694f0b8d0d69012
May 20 15:42:14.954397 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:14.954353    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:15.104763 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.104736    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"]
May 20 15:42:15.105287 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:42:15.105261    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efd0743_d1ca_4cbd_a724_c18df12d657e.slice/crio-a5ae2958b84a18e83c15e24a7a9e4d8332c236b9df2c31b0b4ee65a49834c6d9 WatchSource:0}: Error finding container a5ae2958b84a18e83c15e24a7a9e4d8332c236b9df2c31b0b4ee65a49834c6d9: Status 404 returned error can't find the container with id a5ae2958b84a18e83c15e24a7a9e4d8332c236b9df2c31b0b4ee65a49834c6d9
May 20 15:42:15.518162 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.518130    2567 generic.go:358] "Generic (PLEG): container finished" podID="87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" containerID="1e0a23a5b7e86de945965fb800dd0a32cea860b038a5bd367f8283ad3f625696" exitCode=0
May 20 15:42:15.518603 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.518215    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf" event={"ID":"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb","Type":"ContainerDied","Data":"1e0a23a5b7e86de945965fb800dd0a32cea860b038a5bd367f8283ad3f625696"}
May 20 15:42:15.518603 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.518257    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf" event={"ID":"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb","Type":"ContainerStarted","Data":"58b7cda2f51e60adeaecc1f2806ff750fa6ae1f7c4a8e3957694f0b8d0d69012"}
May 20 15:42:15.519839 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.519815    2567 generic.go:358] "Generic (PLEG): container finished" podID="4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" containerID="9b439e34132536a54716735ee2e8ed3cf06d7e715ea5f508c20cab2fc85554a3" exitCode=0
May 20 15:42:15.519930 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.519901    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7" event={"ID":"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c","Type":"ContainerDied","Data":"9b439e34132536a54716735ee2e8ed3cf06d7e715ea5f508c20cab2fc85554a3"}
May 20 15:42:15.519981 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.519937    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7" event={"ID":"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c","Type":"ContainerStarted","Data":"c09c76a2923be8e95ca5dfd7abd30ae57da415fb8693158c99c12e83a3fac20b"}
May 20 15:42:15.521496 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.521364    2567 generic.go:358] "Generic (PLEG): container finished" podID="8efd0743-d1ca-4cbd-a724-c18df12d657e" containerID="865fc3ffaa4f28aecfaae7a1f280acfa53fe06430f7e795855a6288ac3f70f12" exitCode=0
May 20 15:42:15.521603 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.521501    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2" event={"ID":"8efd0743-d1ca-4cbd-a724-c18df12d657e","Type":"ContainerDied","Data":"865fc3ffaa4f28aecfaae7a1f280acfa53fe06430f7e795855a6288ac3f70f12"}
May 20 15:42:15.521603 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.521530    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2" event={"ID":"8efd0743-d1ca-4cbd-a724-c18df12d657e","Type":"ContainerStarted","Data":"a5ae2958b84a18e83c15e24a7a9e4d8332c236b9df2c31b0b4ee65a49834c6d9"}
May 20 15:42:15.523037 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.523011    2567 generic.go:358] "Generic (PLEG): container finished" podID="63b11480-36cf-48d5-8d89-ec3d78ca39a5" containerID="833d67880aec2f38b8729dc58b53c0c704bf2ff5febf2635895802f2619a20be" exitCode=0
May 20 15:42:15.523165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.523039    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8" event={"ID":"63b11480-36cf-48d5-8d89-ec3d78ca39a5","Type":"ContainerDied","Data":"833d67880aec2f38b8729dc58b53c0c704bf2ff5febf2635895802f2619a20be"}
May 20 15:42:15.523165 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:15.523061    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8" event={"ID":"63b11480-36cf-48d5-8d89-ec3d78ca39a5","Type":"ContainerStarted","Data":"7c6477b5a65a88dece29dfea3ec78a6f6fcbdcd0d45893fc2d9e5219717bb485"}
May 20 15:42:16.529548 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:16.529517    2567 generic.go:358] "Generic (PLEG): container finished" podID="4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" containerID="1dccce50cae4f186a52f6002792d5133efd1388c7083cb402bf2219bfbd4bc34" exitCode=0
May 20 15:42:16.529877 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:16.529624    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7" event={"ID":"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c","Type":"ContainerDied","Data":"1dccce50cae4f186a52f6002792d5133efd1388c7083cb402bf2219bfbd4bc34"}
May 20 15:42:17.534532 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:17.534491    2567 generic.go:358] "Generic (PLEG): container finished" podID="87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" containerID="56d4151cc2ba32336e5dbbd622f9da198bc8de3b73997adc61b88a185ca551e3" exitCode=0
May 20 15:42:17.534991 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:17.534576    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf" event={"ID":"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb","Type":"ContainerDied","Data":"56d4151cc2ba32336e5dbbd622f9da198bc8de3b73997adc61b88a185ca551e3"}
May 20 15:42:17.536609 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:17.536564    2567 generic.go:358] "Generic (PLEG): container finished" podID="4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" containerID="15dac5add3a2f753704a5410c59d05af9e1b96c1f0941ab77e2fcbba61f465da" exitCode=0
May 20 15:42:17.536728 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:17.536615    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7" event={"ID":"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c","Type":"ContainerDied","Data":"15dac5add3a2f753704a5410c59d05af9e1b96c1f0941ab77e2fcbba61f465da"}
May 20 15:42:17.538340 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:17.538316    2567 generic.go:358] "Generic (PLEG): container finished" podID="63b11480-36cf-48d5-8d89-ec3d78ca39a5" containerID="c005e01200b7081abf10319f46f863b98effd2fea48b4a9cb766b0660c5f0159" exitCode=0
May 20 15:42:17.538530 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:17.538377    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8" event={"ID":"63b11480-36cf-48d5-8d89-ec3d78ca39a5","Type":"ContainerDied","Data":"c005e01200b7081abf10319f46f863b98effd2fea48b4a9cb766b0660c5f0159"}
May 20 15:42:18.544426 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.544392    2567 generic.go:358] "Generic (PLEG): container finished" podID="87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" containerID="d22d8f7c6581f351fbb9181058a349cf036382971a9e2efc36d114935495b5ea" exitCode=0
May 20 15:42:18.544816 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.544484    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf" event={"ID":"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb","Type":"ContainerDied","Data":"d22d8f7c6581f351fbb9181058a349cf036382971a9e2efc36d114935495b5ea"}
May 20 15:42:18.546259 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.546239    2567 generic.go:358] "Generic (PLEG): container finished" podID="63b11480-36cf-48d5-8d89-ec3d78ca39a5" containerID="42febd843044bc478bf98dcd4a4257746e2828007ee60023e7e809f42de33d32" exitCode=0
May 20 15:42:18.546354 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.546306    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8" event={"ID":"63b11480-36cf-48d5-8d89-ec3d78ca39a5","Type":"ContainerDied","Data":"42febd843044bc478bf98dcd4a4257746e2828007ee60023e7e809f42de33d32"}
May 20 15:42:18.685379 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.685348    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:18.734528 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.734487    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-bundle\") pod \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") "
May 20 15:42:18.734723 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.734540    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-util\") pod \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") "
May 20 15:42:18.734723 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.734646    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tqwb\" (UniqueName: \"kubernetes.io/projected/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-kube-api-access-2tqwb\") pod \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\" (UID: \"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c\") "
May 20 15:42:18.735118 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.735083    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-bundle" (OuterVolumeSpecName: "bundle") pod "4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" (UID: "4f31b0f9-d7ac-4e86-abe2-010de0a7e76c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:42:18.736846 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.736816    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-kube-api-access-2tqwb" (OuterVolumeSpecName: "kube-api-access-2tqwb") pod "4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" (UID: "4f31b0f9-d7ac-4e86-abe2-010de0a7e76c"). InnerVolumeSpecName "kube-api-access-2tqwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:42:18.740201 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.740175    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-util" (OuterVolumeSpecName: "util") pod "4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" (UID: "4f31b0f9-d7ac-4e86-abe2-010de0a7e76c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:42:18.835421 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.835336    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tqwb\" (UniqueName: \"kubernetes.io/projected/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-kube-api-access-2tqwb\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:18.835421 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.835366    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:18.835421 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:18.835376    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4f31b0f9-d7ac-4e86-abe2-010de0a7e76c-util\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:19.551776 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.551734    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7" event={"ID":"4f31b0f9-d7ac-4e86-abe2-010de0a7e76c","Type":"ContainerDied","Data":"c09c76a2923be8e95ca5dfd7abd30ae57da415fb8693158c99c12e83a3fac20b"}
May 20 15:42:19.551776 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.551779    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c09c76a2923be8e95ca5dfd7abd30ae57da415fb8693158c99c12e83a3fac20b"
May 20 15:42:19.552197 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.551800    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zr2x7"
May 20 15:42:19.740336 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.740309    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:19.743971 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.743950    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:19.844734 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.844711    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-util\") pod \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") "
May 20 15:42:19.844829 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.844742    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-bundle\") pod \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") "
May 20 15:42:19.844829 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.844778    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9b7r\" (UniqueName: \"kubernetes.io/projected/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-kube-api-access-k9b7r\") pod \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") "
May 20 15:42:19.844829 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.844819    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-bundle\") pod \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") "
May 20 15:42:19.845002 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.844844    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdlzm\" (UniqueName: \"kubernetes.io/projected/63b11480-36cf-48d5-8d89-ec3d78ca39a5-kube-api-access-tdlzm\") pod \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\" (UID: \"63b11480-36cf-48d5-8d89-ec3d78ca39a5\") "
May 20 15:42:19.845002 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.844862    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-util\") pod \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\" (UID: \"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb\") "
May 20 15:42:19.845565 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.845502    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-bundle" (OuterVolumeSpecName: "bundle") pod "63b11480-36cf-48d5-8d89-ec3d78ca39a5" (UID: "63b11480-36cf-48d5-8d89-ec3d78ca39a5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:42:19.845699 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.845629    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-bundle" (OuterVolumeSpecName: "bundle") pod "87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" (UID: "87eba398-ee9a-4623-8ba4-18ee9bf9d9fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:42:19.847711 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.847677    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-kube-api-access-k9b7r" (OuterVolumeSpecName: "kube-api-access-k9b7r") pod "87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" (UID: "87eba398-ee9a-4623-8ba4-18ee9bf9d9fb"). InnerVolumeSpecName "kube-api-access-k9b7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:42:19.847826 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.847731    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b11480-36cf-48d5-8d89-ec3d78ca39a5-kube-api-access-tdlzm" (OuterVolumeSpecName: "kube-api-access-tdlzm") pod "63b11480-36cf-48d5-8d89-ec3d78ca39a5" (UID: "63b11480-36cf-48d5-8d89-ec3d78ca39a5"). InnerVolumeSpecName "kube-api-access-tdlzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:42:19.851729 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.851691    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-util" (OuterVolumeSpecName: "util") pod "63b11480-36cf-48d5-8d89-ec3d78ca39a5" (UID: "63b11480-36cf-48d5-8d89-ec3d78ca39a5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:42:19.851856 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.851801    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-util" (OuterVolumeSpecName: "util") pod "87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" (UID: "87eba398-ee9a-4623-8ba4-18ee9bf9d9fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:42:19.945904 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.945860    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-util\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:19.945904 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.945900    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63b11480-36cf-48d5-8d89-ec3d78ca39a5-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:19.946077 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.945916    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9b7r\" (UniqueName: \"kubernetes.io/projected/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-kube-api-access-k9b7r\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:19.946077 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.945928    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:19.946077 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.945937    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlzm\" (UniqueName: \"kubernetes.io/projected/63b11480-36cf-48d5-8d89-ec3d78ca39a5-kube-api-access-tdlzm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:19.946077 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:19.945945    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87eba398-ee9a-4623-8ba4-18ee9bf9d9fb-util\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:20.557326 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:20.557285    2567 generic.go:358] "Generic (PLEG): container finished" podID="8efd0743-d1ca-4cbd-a724-c18df12d657e" containerID="1620c846593e17278cf8aacd96dc45930bd05f2fbc092680f454c082dc633f9a" exitCode=0
May 20 15:42:20.557843 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:20.557364    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2" event={"ID":"8efd0743-d1ca-4cbd-a724-c18df12d657e","Type":"ContainerDied","Data":"1620c846593e17278cf8aacd96dc45930bd05f2fbc092680f454c082dc633f9a"}
May 20 15:42:20.559285 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:20.559261    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8" event={"ID":"63b11480-36cf-48d5-8d89-ec3d78ca39a5","Type":"ContainerDied","Data":"7c6477b5a65a88dece29dfea3ec78a6f6fcbdcd0d45893fc2d9e5219717bb485"}
May 20 15:42:20.559366 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:20.559296    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c6477b5a65a88dece29dfea3ec78a6f6fcbdcd0d45893fc2d9e5219717bb485"
May 20 15:42:20.559366 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:20.559264    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30bw5b8"
May 20 15:42:20.561083 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:20.561057    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf" event={"ID":"87eba398-ee9a-4623-8ba4-18ee9bf9d9fb","Type":"ContainerDied","Data":"58b7cda2f51e60adeaecc1f2806ff750fa6ae1f7c4a8e3957694f0b8d0d69012"}
May 20 15:42:20.561083 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:20.561078    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78dpfnf"
May 20 15:42:20.561215 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:20.561087    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b7cda2f51e60adeaecc1f2806ff750fa6ae1f7c4a8e3957694f0b8d0d69012"
May 20 15:42:21.566920 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:21.566885    2567 generic.go:358] "Generic (PLEG): container finished" podID="8efd0743-d1ca-4cbd-a724-c18df12d657e" containerID="249b75b9b9c88f32ee9f1c00cb135e08efd868ef88dbb7fb0a74665b37dda08f" exitCode=0
May 20 15:42:21.567290 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:21.566979    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2" event={"ID":"8efd0743-d1ca-4cbd-a724-c18df12d657e","Type":"ContainerDied","Data":"249b75b9b9c88f32ee9f1c00cb135e08efd868ef88dbb7fb0a74665b37dda08f"}
May 20 15:42:22.703331 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:22.703305    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:22.769436 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:22.769401    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd69z\" (UniqueName: \"kubernetes.io/projected/8efd0743-d1ca-4cbd-a724-c18df12d657e-kube-api-access-pd69z\") pod \"8efd0743-d1ca-4cbd-a724-c18df12d657e\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") "
May 20 15:42:22.769683 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:22.769467    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-util\") pod \"8efd0743-d1ca-4cbd-a724-c18df12d657e\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") "
May 20 15:42:22.769683 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:22.769490    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-bundle\") pod \"8efd0743-d1ca-4cbd-a724-c18df12d657e\" (UID: \"8efd0743-d1ca-4cbd-a724-c18df12d657e\") "
May 20 15:42:22.769978 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:22.769950    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-bundle" (OuterVolumeSpecName: "bundle") pod "8efd0743-d1ca-4cbd-a724-c18df12d657e" (UID: "8efd0743-d1ca-4cbd-a724-c18df12d657e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:42:22.771845 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:22.771816    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efd0743-d1ca-4cbd-a724-c18df12d657e-kube-api-access-pd69z" (OuterVolumeSpecName: "kube-api-access-pd69z") pod "8efd0743-d1ca-4cbd-a724-c18df12d657e" (UID: "8efd0743-d1ca-4cbd-a724-c18df12d657e"). InnerVolumeSpecName "kube-api-access-pd69z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:42:22.773997 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:22.773965    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-util" (OuterVolumeSpecName: "util") pod "8efd0743-d1ca-4cbd-a724-c18df12d657e" (UID: "8efd0743-d1ca-4cbd-a724-c18df12d657e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:42:22.870738 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:22.870700    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd69z\" (UniqueName: \"kubernetes.io/projected/8efd0743-d1ca-4cbd-a724-c18df12d657e-kube-api-access-pd69z\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:22.870738 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:22.870734    2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-util\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:22.870738 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:22.870745    2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8efd0743-d1ca-4cbd-a724-c18df12d657e-bundle\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:42:23.576909 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:23.576879    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2"
May 20 15:42:23.577061 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:23.576881    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bjskq2" event={"ID":"8efd0743-d1ca-4cbd-a724-c18df12d657e","Type":"ContainerDied","Data":"a5ae2958b84a18e83c15e24a7a9e4d8332c236b9df2c31b0b4ee65a49834c6d9"}
May 20 15:42:23.577061 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:23.576985    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5ae2958b84a18e83c15e24a7a9e4d8332c236b9df2c31b0b4ee65a49834c6d9"
May 20 15:42:36.421489 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.421445    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp"]
May 20 15:42:36.422102 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422041    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8efd0743-d1ca-4cbd-a724-c18df12d657e" containerName="pull"
May 20 15:42:36.422102 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422062    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8efd0743-d1ca-4cbd-a724-c18df12d657e" containerName="pull"
May 20 15:42:36.422102 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422090    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" containerName="util"
May 20 15:42:36.422102 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422100    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" containerName="util"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422109    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8efd0743-d1ca-4cbd-a724-c18df12d657e" containerName="extract"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422117    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8efd0743-d1ca-4cbd-a724-c18df12d657e" containerName="extract"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422128    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" containerName="pull"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422136    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" containerName="pull"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422148    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" containerName="pull"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422156    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" containerName="pull"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422166    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" containerName="extract"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422174    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" containerName="extract"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422187    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8efd0743-d1ca-4cbd-a724-c18df12d657e" containerName="util"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422196    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8efd0743-d1ca-4cbd-a724-c18df12d657e" containerName="util"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422210    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b11480-36cf-48d5-8d89-ec3d78ca39a5" containerName="util"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422219    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b11480-36cf-48d5-8d89-ec3d78ca39a5" containerName="util"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422229    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b11480-36cf-48d5-8d89-ec3d78ca39a5" containerName="pull"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422238    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b11480-36cf-48d5-8d89-ec3d78ca39a5" containerName="pull"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422247    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" containerName="extract"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422255    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" containerName="extract"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422271    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" containerName="util"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422279    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" containerName="util"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422288    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b11480-36cf-48d5-8d89-ec3d78ca39a5" containerName="extract"
May 20 15:42:36.422314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422296    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b11480-36cf-48d5-8d89-ec3d78ca39a5" containerName="extract"
May 20 15:42:36.423242 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422383    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="8efd0743-d1ca-4cbd-a724-c18df12d657e" containerName="extract"
May 20 15:42:36.423242 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422398    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="4f31b0f9-d7ac-4e86-abe2-010de0a7e76c" containerName="extract"
May 20 15:42:36.423242 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422409    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="87eba398-ee9a-4623-8ba4-18ee9bf9d9fb" containerName="extract"
May 20 15:42:36.423242 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.422418    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="63b11480-36cf-48d5-8d89-ec3d78ca39a5" containerName="extract"
May 20 15:42:36.424699 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.424677    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp"
May 20 15:42:36.427671 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.427639    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 20 15:42:36.427798 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.427685    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 20 15:42:36.428831 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.428809    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-684hm\""
May 20 15:42:36.435869 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.435841    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp"]
May 20 15:42:36.501239 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.501208    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs64p\" (UniqueName: \"kubernetes.io/projected/f051a663-4c2a-4524-803e-fc68ffdbc8a9-kube-api-access-qs64p\") pod \"limitador-operator-controller-manager-c7fb4c8d5-bgqqp\" (UID: \"f051a663-4c2a-4524-803e-fc68ffdbc8a9\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp"
May 20 15:42:36.601894 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.601854    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qs64p\" (UniqueName: \"kubernetes.io/projected/f051a663-4c2a-4524-803e-fc68ffdbc8a9-kube-api-access-qs64p\") pod \"limitador-operator-controller-manager-c7fb4c8d5-bgqqp\" (UID: \"f051a663-4c2a-4524-803e-fc68ffdbc8a9\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp"
May 20 15:42:36.619412 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.619387    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs64p\" (UniqueName: \"kubernetes.io/projected/f051a663-4c2a-4524-803e-fc68ffdbc8a9-kube-api-access-qs64p\") pod \"limitador-operator-controller-manager-c7fb4c8d5-bgqqp\" (UID: \"f051a663-4c2a-4524-803e-fc68ffdbc8a9\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp"
May 20 15:42:36.735899 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.735805    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp"
May 20 15:42:36.882000 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:36.881973    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp"]
May 20 15:42:36.883259 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:42:36.883230    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf051a663_4c2a_4524_803e_fc68ffdbc8a9.slice/crio-5d6def91b8ea526abe6b0d5447468fc062ce4a2cc769a558f99f1c34f718ce2e WatchSource:0}: Error finding container 5d6def91b8ea526abe6b0d5447468fc062ce4a2cc769a558f99f1c34f718ce2e: Status 404 returned error can't find the container with id 5d6def91b8ea526abe6b0d5447468fc062ce4a2cc769a558f99f1c34f718ce2e
May 20 15:42:37.640765 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:37.640726    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp" event={"ID":"f051a663-4c2a-4524-803e-fc68ffdbc8a9","Type":"ContainerStarted","Data":"5d6def91b8ea526abe6b0d5447468fc062ce4a2cc769a558f99f1c34f718ce2e"}
May 20 15:42:41.658144 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:41.658101    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp" event={"ID":"f051a663-4c2a-4524-803e-fc68ffdbc8a9","Type":"ContainerStarted","Data":"19761f978efa80a01593cf152adccedcdcb03265005c2eb24ac690117988943e"}
May 20 15:42:41.658574 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:41.658165    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp"
May 20 15:42:41.694519 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:41.694463    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp" podStartSLOduration=1.5925542369999999 podStartE2EDuration="5.694447212s" podCreationTimestamp="2026-05-20 15:42:36 +0000 UTC" firstStartedPulling="2026-05-20 15:42:36.885228453 +0000 UTC m=+646.589322021" lastFinishedPulling="2026-05-20 15:42:40.987121428 +0000 UTC m=+650.691214996" observedRunningTime="2026-05-20 15:42:41.693797871 +0000 UTC m=+651.397891464" watchObservedRunningTime="2026-05-20 15:42:41.694447212 +0000 UTC m=+651.398540801"
May 20 15:42:44.365820 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:44.365779    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7587b89b76-h5ll8"]
May 20 15:42:44.371075 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:44.371053    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7587b89b76-h5ll8"
May 20 15:42:44.374811 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:44.374790    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rhfd2\""
May 20 15:42:44.389141 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:44.389114    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7587b89b76-h5ll8"]
May 20 15:42:44.486691 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:44.486657    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bmqg\" (UniqueName: \"kubernetes.io/projected/b19a1204-3297-4a76-a566-7ba97481abd0-kube-api-access-7bmqg\") pod \"authorino-operator-7587b89b76-h5ll8\" (UID: \"b19a1204-3297-4a76-a566-7ba97481abd0\") " pod="kuadrant-system/authorino-operator-7587b89b76-h5ll8"
May 20 15:42:44.588122 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:44.588066    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7bmqg\" (UniqueName: \"kubernetes.io/projected/b19a1204-3297-4a76-a566-7ba97481abd0-kube-api-access-7bmqg\") pod \"authorino-operator-7587b89b76-h5ll8\" (UID: \"b19a1204-3297-4a76-a566-7ba97481abd0\") " pod="kuadrant-system/authorino-operator-7587b89b76-h5ll8"
May 20 15:42:44.599359 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:44.599327    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bmqg\" (UniqueName: \"kubernetes.io/projected/b19a1204-3297-4a76-a566-7ba97481abd0-kube-api-access-7bmqg\") pod \"authorino-operator-7587b89b76-h5ll8\" (UID: \"b19a1204-3297-4a76-a566-7ba97481abd0\") " pod="kuadrant-system/authorino-operator-7587b89b76-h5ll8"
May 20 15:42:44.681840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:44.681740    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7587b89b76-h5ll8"
May 20 15:42:44.831309 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:44.831254    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7587b89b76-h5ll8"]
May 20 15:42:44.837190 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:42:44.837159    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19a1204_3297_4a76_a566_7ba97481abd0.slice/crio-e0c2a8a5783f9ec6e1cf2d52bfa8f4b27a95d342fc0bca47068f324056176f2b WatchSource:0}: Error finding container e0c2a8a5783f9ec6e1cf2d52bfa8f4b27a95d342fc0bca47068f324056176f2b: Status 404 returned error can't find the container with id e0c2a8a5783f9ec6e1cf2d52bfa8f4b27a95d342fc0bca47068f324056176f2b
May 20 15:42:45.675667 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:45.675625    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7587b89b76-h5ll8" event={"ID":"b19a1204-3297-4a76-a566-7ba97481abd0","Type":"ContainerStarted","Data":"e0c2a8a5783f9ec6e1cf2d52bfa8f4b27a95d342fc0bca47068f324056176f2b"}
May 20 15:42:47.685839 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:47.685799    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7587b89b76-h5ll8" event={"ID":"b19a1204-3297-4a76-a566-7ba97481abd0","Type":"ContainerStarted","Data":"a94ebfc67efb18daf086e33a63452fd2ec7972339fb4ce344fe47cd81f6e7a85"}
May 20 15:42:47.686265 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:47.685910    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7587b89b76-h5ll8"
May 20 15:42:47.732763 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:47.732711    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7587b89b76-h5ll8" podStartSLOduration=1.941335517 podStartE2EDuration="3.73269403s" podCreationTimestamp="2026-05-20 15:42:44 +0000 UTC" firstStartedPulling="2026-05-20 15:42:44.83985196 +0000 UTC m=+654.543945528" lastFinishedPulling="2026-05-20 15:42:46.63121047 +0000 UTC m=+656.335304041" observedRunningTime="2026-05-20 15:42:47.730537788 +0000 UTC m=+657.434631378" watchObservedRunningTime="2026-05-20 15:42:47.73269403 +0000 UTC m=+657.436787620"
May 20 15:42:52.664854 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:52.664824    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-bgqqp"
May 20 15:42:58.692039 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:42:58.692003    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7587b89b76-h5ll8"
May 20 15:43:29.878056 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:29.878020    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-2svzj"]
May 20 15:43:29.886901 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:29.886874    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj"
May 20 15:43:29.891638 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:29.891612    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 20 15:43:29.892256 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:29.892229    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gnrgp\""
May 20 15:43:29.908172 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:29.908132    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-2svzj"]
May 20 15:43:29.938946 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:29.938908    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-2svzj"]
May 20 15:43:30.000691 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:30.000648    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd4w5\" (UniqueName: \"kubernetes.io/projected/622c88fc-eecb-4d0a-b074-aca2f27ca4f0-kube-api-access-jd4w5\") pod \"limitador-limitador-67566c68b4-2svzj\" (UID: \"622c88fc-eecb-4d0a-b074-aca2f27ca4f0\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj"
May 20 15:43:30.000862 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:30.000792    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/622c88fc-eecb-4d0a-b074-aca2f27ca4f0-config-file\") pod \"limitador-limitador-67566c68b4-2svzj\" (UID: \"622c88fc-eecb-4d0a-b074-aca2f27ca4f0\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj"
May 20 15:43:30.101530 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:30.101479    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/622c88fc-eecb-4d0a-b074-aca2f27ca4f0-config-file\") pod \"limitador-limitador-67566c68b4-2svzj\" (UID: \"622c88fc-eecb-4d0a-b074-aca2f27ca4f0\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj"
May 20 15:43:30.101740 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:30.101546    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4w5\" (UniqueName: \"kubernetes.io/projected/622c88fc-eecb-4d0a-b074-aca2f27ca4f0-kube-api-access-jd4w5\") pod \"limitador-limitador-67566c68b4-2svzj\" (UID: \"622c88fc-eecb-4d0a-b074-aca2f27ca4f0\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj"
May 20 15:43:30.102154 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:30.102134    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/622c88fc-eecb-4d0a-b074-aca2f27ca4f0-config-file\") pod \"limitador-limitador-67566c68b4-2svzj\" (UID: \"622c88fc-eecb-4d0a-b074-aca2f27ca4f0\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj"
May 20 15:43:30.114794 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:30.114760    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd4w5\" (UniqueName: \"kubernetes.io/projected/622c88fc-eecb-4d0a-b074-aca2f27ca4f0-kube-api-access-jd4w5\") pod \"limitador-limitador-67566c68b4-2svzj\" (UID: \"622c88fc-eecb-4d0a-b074-aca2f27ca4f0\") " pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj"
May 20 15:43:30.198419 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:30.198323    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj"
May 20 15:43:30.343813 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:30.343781    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-2svzj"]
May 20 15:43:30.344967 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:43:30.344942    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622c88fc_eecb_4d0a_b074_aca2f27ca4f0.slice/crio-b1eacc99eedefd41e613b0ca2d14ec77a5353044639b423e32bdcc04cbd43551 WatchSource:0}: Error finding container b1eacc99eedefd41e613b0ca2d14ec77a5353044639b423e32bdcc04cbd43551: Status 404 returned error can't find the container with id b1eacc99eedefd41e613b0ca2d14ec77a5353044639b423e32bdcc04cbd43551
May 20 15:43:30.346829 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:30.346812    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:43:30.871570 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:30.871531    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj" event={"ID":"622c88fc-eecb-4d0a-b074-aca2f27ca4f0","Type":"ContainerStarted","Data":"b1eacc99eedefd41e613b0ca2d14ec77a5353044639b423e32bdcc04cbd43551"}
May 20 15:43:35.893038 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:35.892998    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj" event={"ID":"622c88fc-eecb-4d0a-b074-aca2f27ca4f0","Type":"ContainerStarted","Data":"cc824da06faf808843ed959739b799abd2ad69979ab33951d6282e379573824b"}
May 20 15:43:35.893459 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:35.893188    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj"
May 20 15:43:35.915912 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:35.915844    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj" podStartSLOduration=2.093521695 podStartE2EDuration="6.915825345s" podCreationTimestamp="2026-05-20 15:43:29 +0000 UTC" firstStartedPulling="2026-05-20 15:43:30.346936583 +0000 UTC m=+700.051030151" lastFinishedPulling="2026-05-20 15:43:35.169240219 +0000 UTC m=+704.873333801" observedRunningTime="2026-05-20 15:43:35.914201165 +0000 UTC m=+705.618294755" watchObservedRunningTime="2026-05-20 15:43:35.915825345 +0000 UTC m=+705.619918981"
May 20 15:43:46.899332 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:43:46.899298    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-67566c68b4-2svzj"
May 20 15:44:04.532103 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.532061    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-d2s9d"]
May 20 15:44:04.532577 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.532294    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d" podUID="5c200523-580a-4852-9cff-0318b950ca82" containerName="manager" containerID="cri-o://c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35" gracePeriod=10
May 20 15:44:04.563825 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.563789    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-p89ct"]
May 20 15:44:04.567155 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.567139    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-p89ct"
May 20 15:44:04.579740 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.579696    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-p89ct"]
May 20 15:44:04.620170 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.620132    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e549444-3b41-44b3-bc39-669145afbabd-cert\") pod \"kserve-controller-manager-87b4b9896-p89ct\" (UID: \"7e549444-3b41-44b3-bc39-669145afbabd\") " pod="kserve/kserve-controller-manager-87b4b9896-p89ct"
May 20 15:44:04.620327 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.620186    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt9rd\" (UniqueName: \"kubernetes.io/projected/7e549444-3b41-44b3-bc39-669145afbabd-kube-api-access-jt9rd\") pod \"kserve-controller-manager-87b4b9896-p89ct\" (UID: \"7e549444-3b41-44b3-bc39-669145afbabd\") " pod="kserve/kserve-controller-manager-87b4b9896-p89ct"
May 20 15:44:04.721194 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.721162    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jt9rd\" (UniqueName: \"kubernetes.io/projected/7e549444-3b41-44b3-bc39-669145afbabd-kube-api-access-jt9rd\") pod \"kserve-controller-manager-87b4b9896-p89ct\" (UID: \"7e549444-3b41-44b3-bc39-669145afbabd\") " pod="kserve/kserve-controller-manager-87b4b9896-p89ct"
May 20 15:44:04.721346 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.721255    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e549444-3b41-44b3-bc39-669145afbabd-cert\") pod \"kserve-controller-manager-87b4b9896-p89ct\" (UID: \"7e549444-3b41-44b3-bc39-669145afbabd\") " pod="kserve/kserve-controller-manager-87b4b9896-p89ct"
May 20 15:44:04.724254 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.724224    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e549444-3b41-44b3-bc39-669145afbabd-cert\") pod \"kserve-controller-manager-87b4b9896-p89ct\" (UID: \"7e549444-3b41-44b3-bc39-669145afbabd\") " pod="kserve/kserve-controller-manager-87b4b9896-p89ct"
May 20 15:44:04.732534 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.732476    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt9rd\" (UniqueName: \"kubernetes.io/projected/7e549444-3b41-44b3-bc39-669145afbabd-kube-api-access-jt9rd\") pod \"kserve-controller-manager-87b4b9896-p89ct\" (UID: \"7e549444-3b41-44b3-bc39-669145afbabd\") " pod="kserve/kserve-controller-manager-87b4b9896-p89ct"
May 20 15:44:04.789121 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.789056    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:44:04.913988 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.913953    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-p89ct"
May 20 15:44:04.923138 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.923108    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c200523-580a-4852-9cff-0318b950ca82-cert\") pod \"5c200523-580a-4852-9cff-0318b950ca82\" (UID: \"5c200523-580a-4852-9cff-0318b950ca82\") "
May 20 15:44:04.923273 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.923241    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf2lc\" (UniqueName: \"kubernetes.io/projected/5c200523-580a-4852-9cff-0318b950ca82-kube-api-access-pf2lc\") pod \"5c200523-580a-4852-9cff-0318b950ca82\" (UID: \"5c200523-580a-4852-9cff-0318b950ca82\") "
May 20 15:44:04.925416 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.925385    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c200523-580a-4852-9cff-0318b950ca82-cert" (OuterVolumeSpecName: "cert") pod "5c200523-580a-4852-9cff-0318b950ca82" (UID: "5c200523-580a-4852-9cff-0318b950ca82"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:44:04.925416 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:04.925398    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c200523-580a-4852-9cff-0318b950ca82-kube-api-access-pf2lc" (OuterVolumeSpecName: "kube-api-access-pf2lc") pod "5c200523-580a-4852-9cff-0318b950ca82" (UID: "5c200523-580a-4852-9cff-0318b950ca82"). InnerVolumeSpecName "kube-api-access-pf2lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:44:05.012221 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.012188    2567 generic.go:358] "Generic (PLEG): container finished" podID="5c200523-580a-4852-9cff-0318b950ca82" containerID="c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35" exitCode=0
May 20 15:44:05.012389 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.012266    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d"
May 20 15:44:05.012389 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.012267    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d" event={"ID":"5c200523-580a-4852-9cff-0318b950ca82","Type":"ContainerDied","Data":"c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35"}
May 20 15:44:05.012389 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.012365    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-d2s9d" event={"ID":"5c200523-580a-4852-9cff-0318b950ca82","Type":"ContainerDied","Data":"0a3e6c7d6e24f3300513bdc45062b750f7e4f0aa4e8bd2c40657b61dc27d2f4a"}
May 20 15:44:05.012389 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.012380    2567 scope.go:117] "RemoveContainer" containerID="c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35"
May 20 15:44:05.025549 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.025524    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf2lc\" (UniqueName: \"kubernetes.io/projected/5c200523-580a-4852-9cff-0318b950ca82-kube-api-access-pf2lc\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:44:05.025764 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.025747    2567 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c200523-580a-4852-9cff-0318b950ca82-cert\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:44:05.027237 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.027216    2567 scope.go:117] "RemoveContainer" containerID="c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35"
May 20 15:44:05.027671 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:44:05.027646    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35\": container with ID starting with c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35 not found: ID does not exist" containerID="c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35"
May 20 15:44:05.027787 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.027680    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35"} err="failed to get container status \"c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35\": rpc error: code = NotFound desc = could not find container \"c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35\": container with ID starting with c36683fa41988c055fd330e7a815a3412c1c80aee8e13dc02c8fb9e12afbef35 not found: ID does not exist"
May 20 15:44:05.044649 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.044538    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-d2s9d"]
May 20 15:44:05.050500 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.050473    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-d2s9d"]
May 20 15:44:05.057146 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:44:05.057118    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e549444_3b41_44b3_bc39_669145afbabd.slice/crio-7edcf1b0091803eaa83f9b31ae2c07148e6154feea664302da10a5add96ee99d WatchSource:0}: Error finding container 7edcf1b0091803eaa83f9b31ae2c07148e6154feea664302da10a5add96ee99d: Status 404 returned error can't find the container with id 7edcf1b0091803eaa83f9b31ae2c07148e6154feea664302da10a5add96ee99d
May 20 15:44:05.059117 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.059095    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c200523-580a-4852-9cff-0318b950ca82" path="/var/lib/kubelet/pods/5c200523-580a-4852-9cff-0318b950ca82/volumes"
May 20 15:44:05.059372 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:05.059360    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-87b4b9896-p89ct"]
May 20 15:44:06.018609 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:06.018552    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-p89ct" event={"ID":"7e549444-3b41-44b3-bc39-669145afbabd","Type":"ContainerStarted","Data":"b73ffc07f2b311b38dc8f959c943ed8cc4047d62ce61dbe81225bbc81cc01f2a"}
May 20 15:44:06.018609 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:06.018613    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-87b4b9896-p89ct"
May 20 15:44:06.019092 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:06.018631    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-87b4b9896-p89ct" event={"ID":"7e549444-3b41-44b3-bc39-669145afbabd","Type":"ContainerStarted","Data":"7edcf1b0091803eaa83f9b31ae2c07148e6154feea664302da10a5add96ee99d"}
May 20 15:44:06.040514 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:06.040458    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-87b4b9896-p89ct" podStartSLOduration=1.664815805 podStartE2EDuration="2.040444049s" podCreationTimestamp="2026-05-20 15:44:04 +0000 UTC" firstStartedPulling="2026-05-20 15:44:05.058253861 +0000 UTC m=+734.762347428" lastFinishedPulling="2026-05-20 15:44:05.433882103 +0000 UTC m=+735.137975672" observedRunningTime="2026-05-20 15:44:06.038304854 +0000 UTC m=+735.742398446" watchObservedRunningTime="2026-05-20 15:44:06.040444049 +0000 UTC m=+735.744537639"
May 20 15:44:37.029951 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:44:37.029914    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-87b4b9896-p89ct"
May 20 15:45:36.427107 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.427070    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"]
May 20 15:45:36.427736 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.427716    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c200523-580a-4852-9cff-0318b950ca82" containerName="manager"
May 20 15:45:36.427815 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.427738    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c200523-580a-4852-9cff-0318b950ca82" containerName="manager"
May 20 15:45:36.427866 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.427841    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="5c200523-580a-4852-9cff-0318b950ca82" containerName="manager"
May 20 15:45:36.431493 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.431468    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.434497 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.434473    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 15:45:36.434656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.434570    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 15:45:36.436061 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.436038    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sxlvb\""
May 20 15:45:36.436163 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.436069    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
May 20 15:45:36.443128 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.443100    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"]
May 20 15:45:36.477856 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.477814    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.477856 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.477851    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.478081 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.477885    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b69sz\" (UniqueName: \"kubernetes.io/projected/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kube-api-access-b69sz\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.478141 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.478083    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.478220 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.478202    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.478265 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.478239    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.478319 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.478288    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.579484 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.579445    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.579484 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.579488    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.579777 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.579531    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.579777 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.579570    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.579777 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.579611    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.579777 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.579656    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b69sz\" (UniqueName: \"kubernetes.io/projected/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kube-api-access-b69sz\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.579777 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.579736    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.580102 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.579973    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.580166 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.580138    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.580528 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.580478    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.580528 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.580486    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.582517 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.582493    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.582772 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.582620    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.589846 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.589815    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b69sz\" (UniqueName: \"kubernetes.io/projected/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kube-api-access-b69sz\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.744863 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.744769    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:36.893304 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:36.893236    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"]
May 20 15:45:36.896028 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:45:36.895996    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30811cf_019d_4a1c_bc55_d7cfdceef7f9.slice/crio-116480b37b683cd8cc39f830773c34253ec7018b08a518856cb1cd87a0a87231 WatchSource:0}: Error finding container 116480b37b683cd8cc39f830773c34253ec7018b08a518856cb1cd87a0a87231: Status 404 returned error can't find the container with id 116480b37b683cd8cc39f830773c34253ec7018b08a518856cb1cd87a0a87231
May 20 15:45:37.405687 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:37.405634    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l" event={"ID":"b30811cf-019d-4a1c-bc55-d7cfdceef7f9","Type":"ContainerStarted","Data":"116480b37b683cd8cc39f830773c34253ec7018b08a518856cb1cd87a0a87231"}
May 20 15:45:40.419713 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:40.419657    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l" event={"ID":"b30811cf-019d-4a1c-bc55-d7cfdceef7f9","Type":"ContainerStarted","Data":"5af4b3a15c16f241b63e6342cc36404291069447b40fc55e82ecfa65449869f9"}
May 20 15:45:44.308147 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.308114    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"]
May 20 15:45:44.308546 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.308390    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l" podUID="b30811cf-019d-4a1c-bc55-d7cfdceef7f9" containerName="storage-initializer" containerID="cri-o://5af4b3a15c16f241b63e6342cc36404291069447b40fc55e82ecfa65449869f9" gracePeriod=30
May 20 15:45:44.438053 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.438023    2567 generic.go:358] "Generic (PLEG): container finished" podID="b30811cf-019d-4a1c-bc55-d7cfdceef7f9" containerID="5af4b3a15c16f241b63e6342cc36404291069447b40fc55e82ecfa65449869f9" exitCode=0
May 20 15:45:44.438220 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.438100    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l" event={"ID":"b30811cf-019d-4a1c-bc55-d7cfdceef7f9","Type":"ContainerDied","Data":"5af4b3a15c16f241b63e6342cc36404291069447b40fc55e82ecfa65449869f9"}
May 20 15:45:44.462951 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.462929    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:44.558408 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.558320    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-model-cache\") pod \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") "
May 20 15:45:44.558408 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.558395    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tls-certs\") pod \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") "
May 20 15:45:44.558624 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.558421    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tmp-dir\") pod \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") "
May 20 15:45:44.558624 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.558470    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b69sz\" (UniqueName: \"kubernetes.io/projected/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kube-api-access-b69sz\") pod \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") "
May 20 15:45:44.558624 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.558511    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kserve-provision-location\") pod \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") "
May 20 15:45:44.558624 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.558565    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-dshm\") pod \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") "
May 20 15:45:44.558624 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.558605    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-home\") pod \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\" (UID: \"b30811cf-019d-4a1c-bc55-d7cfdceef7f9\") "
May 20 15:45:44.558869 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.558668    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "b30811cf-019d-4a1c-bc55-d7cfdceef7f9" (UID: "b30811cf-019d-4a1c-bc55-d7cfdceef7f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:45:44.558869 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.558758    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b30811cf-019d-4a1c-bc55-d7cfdceef7f9" (UID: "b30811cf-019d-4a1c-bc55-d7cfdceef7f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:45:44.558977 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.558875    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-home" (OuterVolumeSpecName: "home") pod "b30811cf-019d-4a1c-bc55-d7cfdceef7f9" (UID: "b30811cf-019d-4a1c-bc55-d7cfdceef7f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:45:44.559031 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.559007    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:45:44.559031 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.559026    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:45:44.559101 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.559039    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:45:44.560876 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.560846    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b30811cf-019d-4a1c-bc55-d7cfdceef7f9" (UID: "b30811cf-019d-4a1c-bc55-d7cfdceef7f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:45:44.560989 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.560874    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kube-api-access-b69sz" (OuterVolumeSpecName: "kube-api-access-b69sz") pod "b30811cf-019d-4a1c-bc55-d7cfdceef7f9" (UID: "b30811cf-019d-4a1c-bc55-d7cfdceef7f9"). InnerVolumeSpecName "kube-api-access-b69sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:45:44.561124 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.561104    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-dshm" (OuterVolumeSpecName: "dshm") pod "b30811cf-019d-4a1c-bc55-d7cfdceef7f9" (UID: "b30811cf-019d-4a1c-bc55-d7cfdceef7f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:45:44.626367 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.626317    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b30811cf-019d-4a1c-bc55-d7cfdceef7f9" (UID: "b30811cf-019d-4a1c-bc55-d7cfdceef7f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:45:44.660090 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.660053    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:45:44.660090 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.660089    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b69sz\" (UniqueName: \"kubernetes.io/projected/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kube-api-access-b69sz\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:45:44.660270 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.660100    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:45:44.660270 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:44.660110    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30811cf-019d-4a1c-bc55-d7cfdceef7f9-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:45:45.443516 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:45.443482    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"
May 20 15:45:45.443516 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:45.443495    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l" event={"ID":"b30811cf-019d-4a1c-bc55-d7cfdceef7f9","Type":"ContainerDied","Data":"116480b37b683cd8cc39f830773c34253ec7018b08a518856cb1cd87a0a87231"}
May 20 15:45:45.444017 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:45.443539    2567 scope.go:117] "RemoveContainer" containerID="5af4b3a15c16f241b63e6342cc36404291069447b40fc55e82ecfa65449869f9"
May 20 15:45:45.485838 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:45.485807    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"]
May 20 15:45:45.493528 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:45.493497    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7cccf8766cmkq8l"]
May 20 15:45:47.058077 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:47.058045    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b30811cf-019d-4a1c-bc55-d7cfdceef7f9" path="/var/lib/kubelet/pods/b30811cf-019d-4a1c-bc55-d7cfdceef7f9/volumes"
May 20 15:45:54.223070 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.222979    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"]
May 20 15:45:54.224045 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.224021    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b30811cf-019d-4a1c-bc55-d7cfdceef7f9" containerName="storage-initializer"
May 20 15:45:54.224191 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.224052    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b30811cf-019d-4a1c-bc55-d7cfdceef7f9" containerName="storage-initializer"
May 20 15:45:54.224257 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.224236    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="b30811cf-019d-4a1c-bc55-d7cfdceef7f9" containerName="storage-initializer"
May 20 15:45:54.230751 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.230446    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.234860 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.234218    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sxlvb\""
May 20 15:45:54.234860 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.234401    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 15:45:54.234860 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.234681    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 15:45:54.235573 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.235415    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
May 20 15:45:54.235573 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.235426    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"]
May 20 15:45:54.355018 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.354975    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.355193 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.355040    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.355193 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.355065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l6pz\" (UniqueName: \"kubernetes.io/projected/22aa6568-a31d-4283-a723-616892f84477-kube-api-access-8l6pz\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.355193 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.355120    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.355193 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.355145    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22aa6568-a31d-4283-a723-616892f84477-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.355193 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.355177    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.355368 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.355237    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.456550 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.456502    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.456550 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.456555    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.456862 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.456629    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.456862 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.456677    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.456862 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.456703    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6pz\" (UniqueName: \"kubernetes.io/projected/22aa6568-a31d-4283-a723-616892f84477-kube-api-access-8l6pz\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.456862 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.456754    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.456862 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.456774    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22aa6568-a31d-4283-a723-616892f84477-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.457126 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.456962    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.457126 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.457076    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.457227 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.457129    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.457280 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.457223    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.459241 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.459213    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.459423 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.459407    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22aa6568-a31d-4283-a723-616892f84477-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.466700 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.466673    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6pz\" (UniqueName: \"kubernetes.io/projected/22aa6568-a31d-4283-a723-616892f84477-kube-api-access-8l6pz\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.545230 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.545124    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:45:54.691454 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:54.691425    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"]
May 20 15:45:54.694306 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:45:54.694271    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22aa6568_a31d_4283_a723_616892f84477.slice/crio-a90c75a8c5841f8d188b5a437a515482db0dedb13bbaa36856d631be885d4346 WatchSource:0}: Error finding container a90c75a8c5841f8d188b5a437a515482db0dedb13bbaa36856d631be885d4346: Status 404 returned error can't find the container with id a90c75a8c5841f8d188b5a437a515482db0dedb13bbaa36856d631be885d4346
May 20 15:45:55.488440 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:55.488403    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr" event={"ID":"22aa6568-a31d-4283-a723-616892f84477","Type":"ContainerStarted","Data":"472b9e4782a404aaa94bc32d0ea5de2644ec97502cf5678b2563704625c9b253"}
May 20 15:45:55.488440 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:55.488443    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr" event={"ID":"22aa6568-a31d-4283-a723-616892f84477","Type":"ContainerStarted","Data":"a90c75a8c5841f8d188b5a437a515482db0dedb13bbaa36856d631be885d4346"}
May 20 15:45:59.507365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:59.507330    2567 generic.go:358] "Generic (PLEG): container finished" podID="22aa6568-a31d-4283-a723-616892f84477" containerID="472b9e4782a404aaa94bc32d0ea5de2644ec97502cf5678b2563704625c9b253" exitCode=0
May 20 15:45:59.507801 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:45:59.507402    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr" event={"ID":"22aa6568-a31d-4283-a723-616892f84477","Type":"ContainerDied","Data":"472b9e4782a404aaa94bc32d0ea5de2644ec97502cf5678b2563704625c9b253"}
May 20 15:46:01.517709 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:01.517672    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr" event={"ID":"22aa6568-a31d-4283-a723-616892f84477","Type":"ContainerStarted","Data":"60ec84946b23df54a94793b37e067ac767bd4f8afbbd4d8124d1e9feb4f3a6d7"}
May 20 15:46:01.539062 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:01.539010    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr" podStartSLOduration=6.221364476 podStartE2EDuration="7.538994897s" podCreationTimestamp="2026-05-20 15:45:54 +0000 UTC" firstStartedPulling="2026-05-20 15:45:59.508504102 +0000 UTC m=+849.212597670" lastFinishedPulling="2026-05-20 15:46:00.826134509 +0000 UTC m=+850.530228091" observedRunningTime="2026-05-20 15:46:01.536378926 +0000 UTC m=+851.240472520" watchObservedRunningTime="2026-05-20 15:46:01.538994897 +0000 UTC m=+851.243088487"
May 20 15:46:04.545597 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:04.545543    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:46:04.545967 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:04.545616    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:46:04.558553 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:04.558520    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:46:05.544999 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:05.544967    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:46:13.830716 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:13.830677    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"]
May 20 15:46:13.831234 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:13.830957    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr" podUID="22aa6568-a31d-4283-a723-616892f84477" containerName="main" containerID="cri-o://60ec84946b23df54a94793b37e067ac767bd4f8afbbd4d8124d1e9feb4f3a6d7" gracePeriod=30
May 20 15:46:14.581680 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.581641    2567 generic.go:358] "Generic (PLEG): container finished" podID="22aa6568-a31d-4283-a723-616892f84477" containerID="60ec84946b23df54a94793b37e067ac767bd4f8afbbd4d8124d1e9feb4f3a6d7" exitCode=0
May 20 15:46:14.581840 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.581721    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr" event={"ID":"22aa6568-a31d-4283-a723-616892f84477","Type":"ContainerDied","Data":"60ec84946b23df54a94793b37e067ac767bd4f8afbbd4d8124d1e9feb4f3a6d7"}
May 20 15:46:14.613447 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.613421    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:46:14.634400 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.634363    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-kserve-provision-location\") pod \"22aa6568-a31d-4283-a723-616892f84477\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") "
May 20 15:46:14.634400 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.634398    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l6pz\" (UniqueName: \"kubernetes.io/projected/22aa6568-a31d-4283-a723-616892f84477-kube-api-access-8l6pz\") pod \"22aa6568-a31d-4283-a723-616892f84477\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") "
May 20 15:46:14.634656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.634460    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-tmp-dir\") pod \"22aa6568-a31d-4283-a723-616892f84477\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") "
May 20 15:46:14.634656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.634507    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22aa6568-a31d-4283-a723-616892f84477-tls-certs\") pod \"22aa6568-a31d-4283-a723-616892f84477\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") "
May 20 15:46:14.634656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.634532    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-model-cache\") pod \"22aa6568-a31d-4283-a723-616892f84477\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") "
May 20 15:46:14.634656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.634553    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-home\") pod \"22aa6568-a31d-4283-a723-616892f84477\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") "
May 20 15:46:14.634656 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.634654    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-dshm\") pod \"22aa6568-a31d-4283-a723-616892f84477\" (UID: \"22aa6568-a31d-4283-a723-616892f84477\") "
May 20 15:46:14.634920 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.634802    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22aa6568-a31d-4283-a723-616892f84477" (UID: "22aa6568-a31d-4283-a723-616892f84477"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:46:14.635051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.635031    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:46:14.635129 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.635103    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-home" (OuterVolumeSpecName: "home") pod "22aa6568-a31d-4283-a723-616892f84477" (UID: "22aa6568-a31d-4283-a723-616892f84477"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:46:14.635193 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.635169    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-model-cache" (OuterVolumeSpecName: "model-cache") pod "22aa6568-a31d-4283-a723-616892f84477" (UID: "22aa6568-a31d-4283-a723-616892f84477"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:46:14.637170 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.637133    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22aa6568-a31d-4283-a723-616892f84477-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22aa6568-a31d-4283-a723-616892f84477" (UID: "22aa6568-a31d-4283-a723-616892f84477"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:46:14.637494 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.637467    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-dshm" (OuterVolumeSpecName: "dshm") pod "22aa6568-a31d-4283-a723-616892f84477" (UID: "22aa6568-a31d-4283-a723-616892f84477"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:46:14.637494 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.637482    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22aa6568-a31d-4283-a723-616892f84477-kube-api-access-8l6pz" (OuterVolumeSpecName: "kube-api-access-8l6pz") pod "22aa6568-a31d-4283-a723-616892f84477" (UID: "22aa6568-a31d-4283-a723-616892f84477"). InnerVolumeSpecName "kube-api-access-8l6pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:46:14.736347 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.736264    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22aa6568-a31d-4283-a723-616892f84477-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:46:14.736347 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.736293    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:46:14.736347 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.736302    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:46:14.736347 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.736309    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:46:14.736347 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:14.736319    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l6pz\" (UniqueName: \"kubernetes.io/projected/22aa6568-a31d-4283-a723-616892f84477-kube-api-access-8l6pz\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:46:15.587148 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:15.587114    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"
May 20 15:46:15.587571 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:15.587119    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr" event={"ID":"22aa6568-a31d-4283-a723-616892f84477","Type":"ContainerDied","Data":"a90c75a8c5841f8d188b5a437a515482db0dedb13bbaa36856d631be885d4346"}
May 20 15:46:15.587571 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:15.587244    2567 scope.go:117] "RemoveContainer" containerID="60ec84946b23df54a94793b37e067ac767bd4f8afbbd4d8124d1e9feb4f3a6d7"
May 20 15:46:15.596072 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:15.596048    2567 scope.go:117] "RemoveContainer" containerID="472b9e4782a404aaa94bc32d0ea5de2644ec97502cf5678b2563704625c9b253"
May 20 15:46:15.974531 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:15.974433    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22aa6568-a31d-4283-a723-616892f84477" (UID: "22aa6568-a31d-4283-a723-616892f84477"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:46:16.046337 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:16.046299    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22aa6568-a31d-4283-a723-616892f84477-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:46:16.210091 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:16.210051    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"]
May 20 15:46:16.213525 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:16.213497    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-696457575br4ctr"]
May 20 15:46:17.057181 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:17.057148    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22aa6568-a31d-4283-a723-616892f84477" path="/var/lib/kubelet/pods/22aa6568-a31d-4283-a723-616892f84477/volumes"
May 20 15:46:51.033155 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:51.033128    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:46:51.035815 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:46:51.035794    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:49:24.727960 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.727917    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"]
May 20 15:49:24.728679 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.728602    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22aa6568-a31d-4283-a723-616892f84477" containerName="main"
May 20 15:49:24.728679 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.728624    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="22aa6568-a31d-4283-a723-616892f84477" containerName="main"
May 20 15:49:24.728679 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.728658    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22aa6568-a31d-4283-a723-616892f84477" containerName="storage-initializer"
May 20 15:49:24.728679 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.728667    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="22aa6568-a31d-4283-a723-616892f84477" containerName="storage-initializer"
May 20 15:49:24.728953 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.728770    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="22aa6568-a31d-4283-a723-616892f84477" containerName="main"
May 20 15:49:24.731197 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.731174    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.734085 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.734040    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 15:49:24.734205 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.734174    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
May 20 15:49:24.734273 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.734251    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sxlvb\""
May 20 15:49:24.735260 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.735241    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-qvscn\""
May 20 15:49:24.735341 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.735269    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 15:49:24.745650 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.745622    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"]
May 20 15:49:24.817688 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.817653    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f190213-0598-40cc-9aa0-2bff88e384ec-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.817893 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.817725    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbw9v\" (UniqueName: \"kubernetes.io/projected/7f190213-0598-40cc-9aa0-2bff88e384ec-kube-api-access-zbw9v\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.817893 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.817750    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.817893 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.817779    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.817893 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.817843    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.818079 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.817894    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.918759 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.918728    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f190213-0598-40cc-9aa0-2bff88e384ec-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.918979 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.918794    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zbw9v\" (UniqueName: \"kubernetes.io/projected/7f190213-0598-40cc-9aa0-2bff88e384ec-kube-api-access-zbw9v\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.918979 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.918818    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.918979 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.918847    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.919151 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.918999    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.919151 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.919050    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.919308 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.919277    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.919423 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.919318    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.919423 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.919396    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.919497 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.919420    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.921461 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.921441    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f190213-0598-40cc-9aa0-2bff88e384ec-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:24.927462 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:24.927439    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbw9v\" (UniqueName: \"kubernetes.io/projected/7f190213-0598-40cc-9aa0-2bff88e384ec-kube-api-access-zbw9v\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:25.041848 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:25.041757    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:25.182731 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:25.182702    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"]
May 20 15:49:25.184371 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:49:25.184337    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f190213_0598_40cc_9aa0_2bff88e384ec.slice/crio-a5d74c720950a09b0a1e600536a801e276b28753c9b65ffe00d1741ab9f25a0f WatchSource:0}: Error finding container a5d74c720950a09b0a1e600536a801e276b28753c9b65ffe00d1741ab9f25a0f: Status 404 returned error can't find the container with id a5d74c720950a09b0a1e600536a801e276b28753c9b65ffe00d1741ab9f25a0f
May 20 15:49:25.186350 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:25.186330    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:49:25.363609 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:25.363542    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" event={"ID":"7f190213-0598-40cc-9aa0-2bff88e384ec","Type":"ContainerStarted","Data":"4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161"}
May 20 15:49:25.363609 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:25.363611    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" event={"ID":"7f190213-0598-40cc-9aa0-2bff88e384ec","Type":"ContainerStarted","Data":"a5d74c720950a09b0a1e600536a801e276b28753c9b65ffe00d1741ab9f25a0f"}
May 20 15:49:26.368577 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:26.368541    2567 generic.go:358] "Generic (PLEG): container finished" podID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerID="4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161" exitCode=0
May 20 15:49:26.369050 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:26.368620    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" event={"ID":"7f190213-0598-40cc-9aa0-2bff88e384ec","Type":"ContainerDied","Data":"4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161"}
May 20 15:49:28.381260 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:28.381222    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" event={"ID":"7f190213-0598-40cc-9aa0-2bff88e384ec","Type":"ContainerStarted","Data":"740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7"}
May 20 15:49:58.529243 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:58.529209    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" event={"ID":"7f190213-0598-40cc-9aa0-2bff88e384ec","Type":"ContainerStarted","Data":"269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0"}
May 20 15:49:58.529740 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:58.529469    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:49:58.532007 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:58.531976    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:49:58.554403 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:58.554352    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podStartSLOduration=2.859728628 podStartE2EDuration="34.554335331s" podCreationTimestamp="2026-05-20 15:49:24 +0000 UTC" firstStartedPulling="2026-05-20 15:49:26.369796427 +0000 UTC m=+1056.073890016" lastFinishedPulling="2026-05-20 15:49:58.064403148 +0000 UTC m=+1087.768496719" observedRunningTime="2026-05-20 15:49:58.552137448 +0000 UTC m=+1088.256231039" watchObservedRunningTime="2026-05-20 15:49:58.554335331 +0000 UTC m=+1088.258428952"
May 20 15:49:59.535402 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:49:59.535361    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:50:05.042372 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:05.042323    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:50:05.042372 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:05.042382    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:50:05.042950 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:05.042717    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.50:8082/healthz\": dial tcp 10.134.0.50:8082: connect: connection refused"
May 20 15:50:05.043822 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:05.043793    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:50:15.043853 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:15.043809    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:50:15.044326 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:15.044038    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:50:15.045379 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:15.045348    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:50:15.045507 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:15.045489    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:50:15.606895 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:15.606856    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:50:25.607692 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:25.607650    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:50:35.607097 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:35.607056    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:50:45.607501 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:45.607456    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:50:55.607084 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:50:55.607031    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:51:05.607144 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:05.607053    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:51:06.543053 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:06.543010    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"]
May 20 15:51:06.543522 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:06.543329    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" containerID="cri-o://740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7" gracePeriod=30
May 20 15:51:06.543522 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:06.543382    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="tokenizer" containerID="cri-o://269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0" gracePeriod=30
May 20 15:51:06.545170 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:06.545120    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 20 15:51:06.826772 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:06.826680    2567 generic.go:358] "Generic (PLEG): container finished" podID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerID="740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7" exitCode=0
May 20 15:51:06.826772 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:06.826756    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" event={"ID":"7f190213-0598-40cc-9aa0-2bff88e384ec","Type":"ContainerDied","Data":"740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7"}
May 20 15:51:08.011256 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.011178    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:51:08.033712 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.033677    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f190213-0598-40cc-9aa0-2bff88e384ec-tls-certs\") pod \"7f190213-0598-40cc-9aa0-2bff88e384ec\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") "
May 20 15:51:08.033883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.033732    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-kserve-provision-location\") pod \"7f190213-0598-40cc-9aa0-2bff88e384ec\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") "
May 20 15:51:08.033883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.033756    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbw9v\" (UniqueName: \"kubernetes.io/projected/7f190213-0598-40cc-9aa0-2bff88e384ec-kube-api-access-zbw9v\") pod \"7f190213-0598-40cc-9aa0-2bff88e384ec\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") "
May 20 15:51:08.033883 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.033808    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-uds\") pod \"7f190213-0598-40cc-9aa0-2bff88e384ec\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") "
May 20 15:51:08.034039 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.034013    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-cache\") pod \"7f190213-0598-40cc-9aa0-2bff88e384ec\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") "
May 20 15:51:08.034081 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.034058    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-tmp\") pod \"7f190213-0598-40cc-9aa0-2bff88e384ec\" (UID: \"7f190213-0598-40cc-9aa0-2bff88e384ec\") "
May 20 15:51:08.034204 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.034055    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "7f190213-0598-40cc-9aa0-2bff88e384ec" (UID: "7f190213-0598-40cc-9aa0-2bff88e384ec"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:51:08.034298 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.034264    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "7f190213-0598-40cc-9aa0-2bff88e384ec" (UID: "7f190213-0598-40cc-9aa0-2bff88e384ec"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:51:08.034627 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.034357    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "7f190213-0598-40cc-9aa0-2bff88e384ec" (UID: "7f190213-0598-40cc-9aa0-2bff88e384ec"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:51:08.034705 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.034602    2567 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-uds\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:51:08.034705 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.034659    2567 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:51:08.034705 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.034675    2567 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-tokenizer-tmp\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:51:08.034876 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.034742    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f190213-0598-40cc-9aa0-2bff88e384ec" (UID: "7f190213-0598-40cc-9aa0-2bff88e384ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:51:08.037009 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.036965    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f190213-0598-40cc-9aa0-2bff88e384ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f190213-0598-40cc-9aa0-2bff88e384ec" (UID: "7f190213-0598-40cc-9aa0-2bff88e384ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:51:08.037177 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.037153    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f190213-0598-40cc-9aa0-2bff88e384ec-kube-api-access-zbw9v" (OuterVolumeSpecName: "kube-api-access-zbw9v") pod "7f190213-0598-40cc-9aa0-2bff88e384ec" (UID: "7f190213-0598-40cc-9aa0-2bff88e384ec"). InnerVolumeSpecName "kube-api-access-zbw9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:51:08.135325 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.135280    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f190213-0598-40cc-9aa0-2bff88e384ec-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:51:08.135325 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.135320    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f190213-0598-40cc-9aa0-2bff88e384ec-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:51:08.135325 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.135337    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbw9v\" (UniqueName: \"kubernetes.io/projected/7f190213-0598-40cc-9aa0-2bff88e384ec-kube-api-access-zbw9v\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:51:08.836875 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.836840    2567 generic.go:358] "Generic (PLEG): container finished" podID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerID="269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0" exitCode=0
May 20 15:51:08.837060 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.836923    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"
May 20 15:51:08.837060 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.836925    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" event={"ID":"7f190213-0598-40cc-9aa0-2bff88e384ec","Type":"ContainerDied","Data":"269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0"}
May 20 15:51:08.837060 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.837041    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d" event={"ID":"7f190213-0598-40cc-9aa0-2bff88e384ec","Type":"ContainerDied","Data":"a5d74c720950a09b0a1e600536a801e276b28753c9b65ffe00d1741ab9f25a0f"}
May 20 15:51:08.837060 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.837057    2567 scope.go:117] "RemoveContainer" containerID="269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0"
May 20 15:51:08.846931 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.846913    2567 scope.go:117] "RemoveContainer" containerID="740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7"
May 20 15:51:08.855826 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.855752    2567 scope.go:117] "RemoveContainer" containerID="4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161"
May 20 15:51:08.860757 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.860731    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"]
May 20 15:51:08.864988 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.864961    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-76cb4d7wz96d"]
May 20 15:51:08.865766 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.865749    2567 scope.go:117] "RemoveContainer" containerID="269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0"
May 20 15:51:08.866079 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:51:08.866061    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0\": container with ID starting with 269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0 not found: ID does not exist" containerID="269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0"
May 20 15:51:08.866150 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.866089    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0"} err="failed to get container status \"269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0\": rpc error: code = NotFound desc = could not find container \"269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0\": container with ID starting with 269fc9d855094f6ee1b245a85cb340264ccd4b1647ac15bbb55846141d2672d0 not found: ID does not exist"
May 20 15:51:08.866150 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.866117    2567 scope.go:117] "RemoveContainer" containerID="740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7"
May 20 15:51:08.866351 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:51:08.866334    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7\": container with ID starting with 740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7 not found: ID does not exist" containerID="740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7"
May 20 15:51:08.866416 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.866358    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7"} err="failed to get container status \"740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7\": rpc error: code = NotFound desc = could not find container \"740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7\": container with ID starting with 740a0c35ba9aa0e471335829a03fae52cd17cd93561a419d96a4f2d504979dd7 not found: ID does not exist"
May 20 15:51:08.866416 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.866377    2567 scope.go:117] "RemoveContainer" containerID="4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161"
May 20 15:51:08.866630 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:51:08.866610    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161\": container with ID starting with 4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161 not found: ID does not exist" containerID="4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161"
May 20 15:51:08.866680 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:08.866636    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161"} err="failed to get container status \"4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161\": rpc error: code = NotFound desc = could not find container \"4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161\": container with ID starting with 4fdc2bd3647e095b1a3eb9657d739cd6a1457b5151cb3f23c6cab07e39c96161 not found: ID does not exist"
May 20 15:51:09.058072 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:09.058027    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" path="/var/lib/kubelet/pods/7f190213-0598-40cc-9aa0-2bff88e384ec/volumes"
May 20 15:51:18.146438 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.146398    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"]
May 20 15:51:18.146955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.146822    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="storage-initializer"
May 20 15:51:18.146955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.146835    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="storage-initializer"
May 20 15:51:18.146955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.146847    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main"
May 20 15:51:18.146955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.146852    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main"
May 20 15:51:18.146955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.146858    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="tokenizer"
May 20 15:51:18.146955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.146865    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="tokenizer"
May 20 15:51:18.146955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.146927    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="main"
May 20 15:51:18.146955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.146937    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="7f190213-0598-40cc-9aa0-2bff88e384ec" containerName="tokenizer"
May 20 15:51:18.150175 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.150145    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.153082 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.153043    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-zrxrq\""
May 20 15:51:18.154261 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.154231    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 15:51:18.154388 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.154295    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 15:51:18.154388 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.154327    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
May 20 15:51:18.154576 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.154556    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sxlvb\""
May 20 15:51:18.173987 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.173946    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"]
May 20 15:51:18.235242 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.235203    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r92z\" (UniqueName: \"kubernetes.io/projected/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kube-api-access-9r92z\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.235242 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.235245    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.235463 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.235275    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.235463 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.235381    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.235463 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.235438    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.235562 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.235491    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.336539 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.336494    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.336755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.336559    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.336755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.336627    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.336755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.336660    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.336755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.336707    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9r92z\" (UniqueName: \"kubernetes.io/projected/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kube-api-access-9r92z\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.336755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.336731    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.337055 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.337029    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.337095 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.337039    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.337134 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.337089    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.337134 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.337126    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.339363 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.339334    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.346946 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.346923    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r92z\" (UniqueName: \"kubernetes.io/projected/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kube-api-access-9r92z\") pod \"precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.462513 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.462404    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:18.610181 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.610145    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"]
May 20 15:51:18.611292 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:51:18.611256    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c4003b_41ff_496d_b7f1_3dd88d4df62a.slice/crio-9e423d9275ddb7267631d77c6d625fa7c71af0d2ccc5e7aa54eaf2b8a80940d4 WatchSource:0}: Error finding container 9e423d9275ddb7267631d77c6d625fa7c71af0d2ccc5e7aa54eaf2b8a80940d4: Status 404 returned error can't find the container with id 9e423d9275ddb7267631d77c6d625fa7c71af0d2ccc5e7aa54eaf2b8a80940d4
May 20 15:51:18.881544 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.881503    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" event={"ID":"d7c4003b-41ff-496d-b7f1-3dd88d4df62a","Type":"ContainerStarted","Data":"d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c"}
May 20 15:51:18.881544 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:18.881545    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" event={"ID":"d7c4003b-41ff-496d-b7f1-3dd88d4df62a","Type":"ContainerStarted","Data":"9e423d9275ddb7267631d77c6d625fa7c71af0d2ccc5e7aa54eaf2b8a80940d4"}
May 20 15:51:19.887187 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:19.887140    2567 generic.go:358] "Generic (PLEG): container finished" podID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerID="d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c" exitCode=0
May 20 15:51:19.887705 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:19.887186    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" event={"ID":"d7c4003b-41ff-496d-b7f1-3dd88d4df62a","Type":"ContainerDied","Data":"d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c"}
May 20 15:51:20.893524 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:20.893489    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" event={"ID":"d7c4003b-41ff-496d-b7f1-3dd88d4df62a","Type":"ContainerStarted","Data":"b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6"}
May 20 15:51:20.893524 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:20.893527    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" event={"ID":"d7c4003b-41ff-496d-b7f1-3dd88d4df62a","Type":"ContainerStarted","Data":"67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007"}
May 20 15:51:20.893966 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:20.893721    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:20.918212 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:20.918145    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" podStartSLOduration=2.91812697 podStartE2EDuration="2.91812697s" podCreationTimestamp="2026-05-20 15:51:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:51:20.914992374 +0000 UTC m=+1170.619085976" watchObservedRunningTime="2026-05-20 15:51:20.91812697 +0000 UTC m=+1170.622220561"
May 20 15:51:28.462871 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:28.462829    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:28.463478 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:28.462886    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:28.464322 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:51:28.464300    2567 logging.go:55] [core] [Channel #74 SubChannel #75]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.51:9003", ServerName: "10.134.0.51:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.51:9003: connect: connection refused"
May 20 15:51:28.465685 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:28.465658    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:28.932968 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:28.932935    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:29.463794 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:29.463732    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.51:9003\" within 1s: context deadline exceeded"
May 20 15:51:38.464091 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:51:38.464056    2567 logging.go:55] [core] [Channel #76 SubChannel #77]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.51:9003", ServerName: "10.134.0.51:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.51:9003: connect: connection refused"
May 20 15:51:39.463834 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:39.463767    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.51:9003\" within 1s: context deadline exceeded"
May 20 15:51:49.937647 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:49.937617    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:51:51.068793 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:51.068760    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:51:51.073804 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:51:51.073781    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:52:19.309963 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:19.309924    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"]
May 20 15:52:19.310622 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:19.310343    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="main" containerID="cri-o://67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007" gracePeriod=30
May 20 15:52:19.310622 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:19.310411    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="tokenizer" containerID="cri-o://b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6" gracePeriod=30
May 20 15:52:19.936395 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:52:19.936354    2567 logging.go:55] [core] [Channel #99 SubChannel #100]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.51:9003", ServerName: "10.134.0.51:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.51:9003: connect: connection refused"
May 20 15:52:20.142799 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:20.142762    2567 generic.go:358] "Generic (PLEG): container finished" podID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerID="67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007" exitCode=0
May 20 15:52:20.142974 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:20.142847    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" event={"ID":"d7c4003b-41ff-496d-b7f1-3dd88d4df62a","Type":"ContainerDied","Data":"67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007"}
May 20 15:52:20.936779 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:20.936728    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.51:9003\" within 1s: context deadline exceeded"
May 20 15:52:20.976772 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:20.976746    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:52:21.135161 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.135123    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-uds\") pod \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") "
May 20 15:52:21.135368 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.135189    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r92z\" (UniqueName: \"kubernetes.io/projected/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kube-api-access-9r92z\") pod \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") "
May 20 15:52:21.135368 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.135241    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-tmp\") pod \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") "
May 20 15:52:21.135368 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.135300    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-cache\") pod \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") "
May 20 15:52:21.135368 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.135343    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kserve-provision-location\") pod \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") "
May 20 15:52:21.135368 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.135367    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tls-certs\") pod \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\" (UID: \"d7c4003b-41ff-496d-b7f1-3dd88d4df62a\") "
May 20 15:52:21.135669 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.135409    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d7c4003b-41ff-496d-b7f1-3dd88d4df62a" (UID: "d7c4003b-41ff-496d-b7f1-3dd88d4df62a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:52:21.135669 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.135657    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "d7c4003b-41ff-496d-b7f1-3dd88d4df62a" (UID: "d7c4003b-41ff-496d-b7f1-3dd88d4df62a"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:52:21.135764 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.135675    2567 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-uds\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:52:21.135805 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.135764    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d7c4003b-41ff-496d-b7f1-3dd88d4df62a" (UID: "d7c4003b-41ff-496d-b7f1-3dd88d4df62a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:52:21.136123 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.136103    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d7c4003b-41ff-496d-b7f1-3dd88d4df62a" (UID: "d7c4003b-41ff-496d-b7f1-3dd88d4df62a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:52:21.137668 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.137639    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kube-api-access-9r92z" (OuterVolumeSpecName: "kube-api-access-9r92z") pod "d7c4003b-41ff-496d-b7f1-3dd88d4df62a" (UID: "d7c4003b-41ff-496d-b7f1-3dd88d4df62a"). InnerVolumeSpecName "kube-api-access-9r92z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:52:21.137821 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.137800    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7c4003b-41ff-496d-b7f1-3dd88d4df62a" (UID: "d7c4003b-41ff-496d-b7f1-3dd88d4df62a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:52:21.149906 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.149875    2567 generic.go:358] "Generic (PLEG): container finished" podID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerID="b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6" exitCode=0
May 20 15:52:21.150051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.149953    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" event={"ID":"d7c4003b-41ff-496d-b7f1-3dd88d4df62a","Type":"ContainerDied","Data":"b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6"}
May 20 15:52:21.150051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.149975    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"
May 20 15:52:21.150051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.149997    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k" event={"ID":"d7c4003b-41ff-496d-b7f1-3dd88d4df62a","Type":"ContainerDied","Data":"9e423d9275ddb7267631d77c6d625fa7c71af0d2ccc5e7aa54eaf2b8a80940d4"}
May 20 15:52:21.150051 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.150014    2567 scope.go:117] "RemoveContainer" containerID="b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6"
May 20 15:52:21.160023 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.159997    2567 scope.go:117] "RemoveContainer" containerID="67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007"
May 20 15:52:21.168606 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.168566    2567 scope.go:117] "RemoveContainer" containerID="d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c"
May 20 15:52:21.176732 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.176699    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"]
May 20 15:52:21.178317 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.178296    2567 scope.go:117] "RemoveContainer" containerID="b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6"
May 20 15:52:21.178880 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:52:21.178850    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6\": container with ID starting with b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6 not found: ID does not exist" containerID="b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6"
May 20 15:52:21.178982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.178919    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6"} err="failed to get container status \"b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6\": rpc error: code = NotFound desc = could not find container \"b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6\": container with ID starting with b192fb079b16b0ca8a1e968155ba3198d19fd5cdb2a433215cc8bb4b8ad63bb6 not found: ID does not exist"
May 20 15:52:21.178982 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.178950    2567 scope.go:117] "RemoveContainer" containerID="67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007"
May 20 15:52:21.180111 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:52:21.179462    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007\": container with ID starting with 67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007 not found: ID does not exist" containerID="67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007"
May 20 15:52:21.180111 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.179506    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007"} err="failed to get container status \"67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007\": rpc error: code = NotFound desc = could not find container \"67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007\": container with ID starting with 67bd5a5c5627d8156f07c7b915a4df17df81f65164f3b180a0dbd42875b5f007 not found: ID does not exist"
May 20 15:52:21.180111 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.179532    2567 scope.go:117] "RemoveContainer" containerID="d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c"
May 20 15:52:21.180111 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:52:21.180095    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c\": container with ID starting with d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c not found: ID does not exist" containerID="d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c"
May 20 15:52:21.180433 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.180124    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c"} err="failed to get container status \"d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c\": rpc error: code = NotFound desc = could not find container \"d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c\": container with ID starting with d7721dd5e8df2a0076adbd0b3463d05ff4d9a717a746f5052d7fcf25a7674d2c not found: ID does not exist"
May 20 15:52:21.182989 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.182968    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-7b74bf5675t5k"]
May 20 15:52:21.236850 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.236809    2567 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-tmp\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:52:21.236850 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.236844    2567 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tokenizer-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:52:21.236850 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.236858    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:52:21.237092 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.236874    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:52:21.237092 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:21.236886    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r92z\" (UniqueName: \"kubernetes.io/projected/d7c4003b-41ff-496d-b7f1-3dd88d4df62a-kube-api-access-9r92z\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:52:23.064267 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:23.064221    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" path="/var/lib/kubelet/pods/d7c4003b-41ff-496d-b7f1-3dd88d4df62a/volumes"
May 20 15:52:33.057170 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.057101    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"]
May 20 15:52:33.057623 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.057526    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="main"
May 20 15:52:33.057623 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.057541    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="main"
May 20 15:52:33.057623 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.057550    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="tokenizer"
May 20 15:52:33.057623 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.057555    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="tokenizer"
May 20 15:52:33.057623 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.057566    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="storage-initializer"
May 20 15:52:33.057623 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.057571    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="storage-initializer"
May 20 15:52:33.057815 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.057675    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="tokenizer"
May 20 15:52:33.057815 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.057684    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="d7c4003b-41ff-496d-b7f1-3dd88d4df62a" containerName="main"
May 20 15:52:33.062231 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.062206    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.070726 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.070698    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sxlvb\""
May 20 15:52:33.070950 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.070698    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 20 15:52:33.071073 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.070754    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 15:52:33.071073 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.070775    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 15:52:33.072802 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.072776    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"]
May 20 15:52:33.153457 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.153423    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlt5d\" (UniqueName: \"kubernetes.io/projected/155f3e29-d932-410b-8805-77552785cf77-kube-api-access-vlt5d\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.153662 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.153476    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-tmp-dir\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.153714 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.153654    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-model-cache\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.153759 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.153725    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f3e29-d932-410b-8805-77552785cf77-tls-certs\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.153804 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.153762    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-dshm\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.153862 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.153808    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-home\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.153915 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.153881    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-kserve-provision-location\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.255337 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.255299    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f3e29-d932-410b-8805-77552785cf77-tls-certs\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.255337 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.255352    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-dshm\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.255633 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.255393    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-home\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.255633 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.255416    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-kserve-provision-location\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.255633 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.255443    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vlt5d\" (UniqueName: \"kubernetes.io/projected/155f3e29-d932-410b-8805-77552785cf77-kube-api-access-vlt5d\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.255633 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.255474    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-tmp-dir\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.255633 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.255519    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-model-cache\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.255949 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.255910    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-home\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.255949 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.255939    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-kserve-provision-location\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.256112 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.255965    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-model-cache\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.256112 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.256034    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-tmp-dir\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.257994 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.257973    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-dshm\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.258105 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.258014    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f3e29-d932-410b-8805-77552785cf77-tls-certs\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.265673 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.265646    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlt5d\" (UniqueName: \"kubernetes.io/projected/155f3e29-d932-410b-8805-77552785cf77-kube-api-access-vlt5d\") pod \"stop-feature-test-kserve-58cd7f6c48-khzfl\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.374983 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.374943    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:52:33.534572 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:33.534538    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"]
May 20 15:52:33.536221 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:52:33.536192    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155f3e29_d932_410b_8805_77552785cf77.slice/crio-73e9648f61d112b6944bba6f2a8ff1dab9fee176b38f8c1dc54425358dd45ffd WatchSource:0}: Error finding container 73e9648f61d112b6944bba6f2a8ff1dab9fee176b38f8c1dc54425358dd45ffd: Status 404 returned error can't find the container with id 73e9648f61d112b6944bba6f2a8ff1dab9fee176b38f8c1dc54425358dd45ffd
May 20 15:52:34.213332 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:34.213285    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" event={"ID":"155f3e29-d932-410b-8805-77552785cf77","Type":"ContainerStarted","Data":"7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200"}
May 20 15:52:34.213769 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:34.213338    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" event={"ID":"155f3e29-d932-410b-8805-77552785cf77","Type":"ContainerStarted","Data":"73e9648f61d112b6944bba6f2a8ff1dab9fee176b38f8c1dc54425358dd45ffd"}
May 20 15:52:41.242894 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:41.242855    2567 generic.go:358] "Generic (PLEG): container finished" podID="155f3e29-d932-410b-8805-77552785cf77" containerID="7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200" exitCode=0
May 20 15:52:41.243345 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:52:41.242933    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" event={"ID":"155f3e29-d932-410b-8805-77552785cf77","Type":"ContainerDied","Data":"7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200"}
May 20 15:53:09.393847 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:53:09.393802    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" event={"ID":"155f3e29-d932-410b-8805-77552785cf77","Type":"ContainerStarted","Data":"92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1"}
May 20 15:53:09.423083 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:53:09.423026    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" podStartSLOduration=9.040849845 podStartE2EDuration="36.423011009s" podCreationTimestamp="2026-05-20 15:52:33 +0000 UTC" firstStartedPulling="2026-05-20 15:52:41.244203993 +0000 UTC m=+1250.948297561" lastFinishedPulling="2026-05-20 15:53:08.626365157 +0000 UTC m=+1278.330458725" observedRunningTime="2026-05-20 15:53:09.418889618 +0000 UTC m=+1279.122983245" watchObservedRunningTime="2026-05-20 15:53:09.423011009 +0000 UTC m=+1279.127104598"
May 20 15:53:13.375489 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:53:13.375427    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:53:13.375489 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:53:13.375497    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:53:13.376969 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:53:13.376938    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
May 20 15:53:23.376183 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:53:23.376131    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
May 20 15:53:33.376272 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:53:33.376227    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
May 20 15:53:43.375647 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:53:43.375556    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
May 20 15:53:53.376062 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:53:53.376018    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
May 20 15:54:03.375933 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:03.375835    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
May 20 15:54:13.376148 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:13.376101    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
May 20 15:54:23.375962 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:23.375913    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
May 20 15:54:33.375824 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:33.375783    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
May 20 15:54:43.385950 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:43.385920    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:54:43.394482 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:43.394451    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:54:44.419225 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:44.419187    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"]
May 20 15:54:44.802110 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:44.802017    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main" containerID="cri-o://92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1" gracePeriod=30
May 20 15:54:59.544657 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.544623    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"]
May 20 15:54:59.547788 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.547755    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.560416 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.560386    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"]
May 20 15:54:59.694705 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.694661    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g74k8\" (UniqueName: \"kubernetes.io/projected/b2b49ba9-1957-47ae-8106-88ce782a61cc-kube-api-access-g74k8\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.694705 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.694714    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-dshm\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.694955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.694784    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b49ba9-1957-47ae-8106-88ce782a61cc-tls-certs\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.694955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.694829    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-home\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.694955 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.694891    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-kserve-provision-location\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.695086 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.694959    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-model-cache\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.695086 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.695023    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-tmp-dir\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.796042 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.795947    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-tmp-dir\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.796042 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.796031    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g74k8\" (UniqueName: \"kubernetes.io/projected/b2b49ba9-1957-47ae-8106-88ce782a61cc-kube-api-access-g74k8\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.796042 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.796067    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-dshm\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.796452 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.796091    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b49ba9-1957-47ae-8106-88ce782a61cc-tls-certs\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.796452 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.796126    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-home\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.796452 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.796169    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-kserve-provision-location\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.796452 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.796216    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-model-cache\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.796744 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.796453    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-tmp-dir\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.796815 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.796780    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-home\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.797164 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.797121    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-model-cache\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.797322 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.797245    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-kserve-provision-location\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.799363 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.799342    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-dshm\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.799505 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.799485    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b49ba9-1957-47ae-8106-88ce782a61cc-tls-certs\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.808777 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.808752    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74k8\" (UniqueName: \"kubernetes.io/projected/b2b49ba9-1957-47ae-8106-88ce782a61cc-kube-api-access-g74k8\") pod \"stop-feature-test-kserve-58cd7f6c48-cwc2f\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:54:59.859317 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:54:59.859287    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:55:00.004720 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:00.004685    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"]
May 20 15:55:00.005977 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:55:00.005943    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b49ba9_1957_47ae_8106_88ce782a61cc.slice/crio-5bc7dfbc7fd798bc3403b5f522b6d031d1dc2cb7ab5b4ec66895b965931a43b9 WatchSource:0}: Error finding container 5bc7dfbc7fd798bc3403b5f522b6d031d1dc2cb7ab5b4ec66895b965931a43b9: Status 404 returned error can't find the container with id 5bc7dfbc7fd798bc3403b5f522b6d031d1dc2cb7ab5b4ec66895b965931a43b9
May 20 15:55:00.007949 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:00.007928    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 15:55:00.871886 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:00.871842    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" event={"ID":"b2b49ba9-1957-47ae-8106-88ce782a61cc","Type":"ContainerStarted","Data":"444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab"}
May 20 15:55:00.872374 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:00.871895    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" event={"ID":"b2b49ba9-1957-47ae-8106-88ce782a61cc","Type":"ContainerStarted","Data":"5bc7dfbc7fd798bc3403b5f522b6d031d1dc2cb7ab5b4ec66895b965931a43b9"}
May 20 15:55:15.089206 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.089178    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-58cd7f6c48-khzfl_155f3e29-d932-410b-8805-77552785cf77/main/0.log"
May 20 15:55:15.089600 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.089527    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:55:15.137804 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.137769    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-home\") pod \"155f3e29-d932-410b-8805-77552785cf77\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") "
May 20 15:55:15.137804 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.137801    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-dshm\") pod \"155f3e29-d932-410b-8805-77552785cf77\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") "
May 20 15:55:15.138046 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.137823    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-kserve-provision-location\") pod \"155f3e29-d932-410b-8805-77552785cf77\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") "
May 20 15:55:15.138046 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.137858    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlt5d\" (UniqueName: \"kubernetes.io/projected/155f3e29-d932-410b-8805-77552785cf77-kube-api-access-vlt5d\") pod \"155f3e29-d932-410b-8805-77552785cf77\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") "
May 20 15:55:15.138046 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.137910    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f3e29-d932-410b-8805-77552785cf77-tls-certs\") pod \"155f3e29-d932-410b-8805-77552785cf77\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") "
May 20 15:55:15.138046 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.137982    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-tmp-dir\") pod \"155f3e29-d932-410b-8805-77552785cf77\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") "
May 20 15:55:15.138046 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.138004    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-model-cache\") pod \"155f3e29-d932-410b-8805-77552785cf77\" (UID: \"155f3e29-d932-410b-8805-77552785cf77\") "
May 20 15:55:15.138703 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.138479    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-model-cache" (OuterVolumeSpecName: "model-cache") pod "155f3e29-d932-410b-8805-77552785cf77" (UID: "155f3e29-d932-410b-8805-77552785cf77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:55:15.138703 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.138551    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-home" (OuterVolumeSpecName: "home") pod "155f3e29-d932-410b-8805-77552785cf77" (UID: "155f3e29-d932-410b-8805-77552785cf77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:55:15.141024 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.140983    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/155f3e29-d932-410b-8805-77552785cf77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "155f3e29-d932-410b-8805-77552785cf77" (UID: "155f3e29-d932-410b-8805-77552785cf77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:55:15.141151 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.141109    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155f3e29-d932-410b-8805-77552785cf77-kube-api-access-vlt5d" (OuterVolumeSpecName: "kube-api-access-vlt5d") pod "155f3e29-d932-410b-8805-77552785cf77" (UID: "155f3e29-d932-410b-8805-77552785cf77"). InnerVolumeSpecName "kube-api-access-vlt5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:55:15.141222 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.140988    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-dshm" (OuterVolumeSpecName: "dshm") pod "155f3e29-d932-410b-8805-77552785cf77" (UID: "155f3e29-d932-410b-8805-77552785cf77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:55:15.151365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.151334    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "155f3e29-d932-410b-8805-77552785cf77" (UID: "155f3e29-d932-410b-8805-77552785cf77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:55:15.207065 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.207001    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "155f3e29-d932-410b-8805-77552785cf77" (UID: "155f3e29-d932-410b-8805-77552785cf77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:55:15.239535 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.239493    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlt5d\" (UniqueName: \"kubernetes.io/projected/155f3e29-d932-410b-8805-77552785cf77-kube-api-access-vlt5d\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:55:15.239535 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.239525    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/155f3e29-d932-410b-8805-77552785cf77-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:55:15.239535 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.239538    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:55:15.239801 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.239550    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:55:15.239801 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.239561    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:55:15.239801 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.239571    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:55:15.239801 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.239602    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155f3e29-d932-410b-8805-77552785cf77-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:55:15.934455 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.934422    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-58cd7f6c48-khzfl_155f3e29-d932-410b-8805-77552785cf77/main/0.log"
May 20 15:55:15.934820 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.934795    2567 generic.go:358] "Generic (PLEG): container finished" podID="155f3e29-d932-410b-8805-77552785cf77" containerID="92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1" exitCode=137
May 20 15:55:15.934885 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.934865    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"
May 20 15:55:15.934885 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.934872    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" event={"ID":"155f3e29-d932-410b-8805-77552785cf77","Type":"ContainerDied","Data":"92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1"}
May 20 15:55:15.934957 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.934903    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl" event={"ID":"155f3e29-d932-410b-8805-77552785cf77","Type":"ContainerDied","Data":"73e9648f61d112b6944bba6f2a8ff1dab9fee176b38f8c1dc54425358dd45ffd"}
May 20 15:55:15.934957 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.934921    2567 scope.go:117] "RemoveContainer" containerID="92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1"
May 20 15:55:15.945011 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.944990    2567 scope.go:117] "RemoveContainer" containerID="7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200"
May 20 15:55:15.955642 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.955617    2567 scope.go:117] "RemoveContainer" containerID="92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1"
May 20 15:55:15.955943 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:55:15.955923    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1\": container with ID starting with 92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1 not found: ID does not exist" containerID="92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1"
May 20 15:55:15.956010 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.955955    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1"} err="failed to get container status \"92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1\": rpc error: code = NotFound desc = could not find container \"92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1\": container with ID starting with 92284ec19a7efd3492736b9eb56884bd31d247e9f40b38a08b579faad86684b1 not found: ID does not exist"
May 20 15:55:15.956010 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.955976    2567 scope.go:117] "RemoveContainer" containerID="7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200"
May 20 15:55:15.956222 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:55:15.956200    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200\": container with ID starting with 7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200 not found: ID does not exist" containerID="7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200"
May 20 15:55:15.956282 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.956236    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200"} err="failed to get container status \"7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200\": rpc error: code = NotFound desc = could not find container \"7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200\": container with ID starting with 7f4416f1e1533ca9181fd006cb48104fbbbff8ad1c83b7bb77af29f137886200 not found: ID does not exist"
May 20 15:55:15.961332 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.961301    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"]
May 20 15:55:15.965755 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:15.965729    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-khzfl"]
May 20 15:55:17.057241 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:55:17.057210    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="155f3e29-d932-410b-8805-77552785cf77" path="/var/lib/kubelet/pods/155f3e29-d932-410b-8805-77552785cf77/volumes"
May 20 15:56:35.250773 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:56:35.250733    2567 generic.go:358] "Generic (PLEG): container finished" podID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerID="444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab" exitCode=0
May 20 15:56:35.251295 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:56:35.250786    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" event={"ID":"b2b49ba9-1957-47ae-8106-88ce782a61cc","Type":"ContainerDied","Data":"444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab"}
May 20 15:56:36.256697 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:56:36.256663    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" event={"ID":"b2b49ba9-1957-47ae-8106-88ce782a61cc","Type":"ContainerStarted","Data":"708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151"}
May 20 15:56:36.280716 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:56:36.280661    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" podStartSLOduration=97.280644311 podStartE2EDuration="1m37.280644311s" podCreationTimestamp="2026-05-20 15:54:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:56:36.276888019 +0000 UTC m=+1485.980981613" watchObservedRunningTime="2026-05-20 15:56:36.280644311 +0000 UTC m=+1485.984737901"
May 20 15:56:39.860449 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:56:39.860404    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:56:39.860933 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:56:39.860465    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:56:39.862038 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:56:39.862006    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
May 20 15:56:49.860547 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:56:49.860502    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
May 20 15:56:51.103438 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:56:51.103409    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:56:51.109980 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:56:51.109955    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 15:56:59.859964 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:56:59.859925    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
May 20 15:57:09.860403 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:57:09.860318    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
May 20 15:57:19.860145 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:57:19.860098    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
May 20 15:57:29.860767 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:57:29.860716    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
May 20 15:57:39.860325 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:57:39.860273    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
May 20 15:57:49.859780 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:57:49.859732    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
May 20 15:57:59.859967 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:57:59.859921    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
May 20 15:58:09.870095 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:09.870059    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:58:09.878334 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:09.878308    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:58:11.080149 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:11.080114    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"]
May 20 15:58:11.648548 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:11.648511    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main" containerID="cri-o://708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151" gracePeriod=30
May 20 15:58:41.927386 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.927362    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-58cd7f6c48-cwc2f_b2b49ba9-1957-47ae-8106-88ce782a61cc/main/0.log"
May 20 15:58:41.927811 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.927754    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:58:41.991341 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.991309    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-dshm\") pod \"b2b49ba9-1957-47ae-8106-88ce782a61cc\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") "
May 20 15:58:41.991341 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.991344    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b49ba9-1957-47ae-8106-88ce782a61cc-tls-certs\") pod \"b2b49ba9-1957-47ae-8106-88ce782a61cc\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") "
May 20 15:58:41.991603 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.991385    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-tmp-dir\") pod \"b2b49ba9-1957-47ae-8106-88ce782a61cc\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") "
May 20 15:58:41.991603 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.991559    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g74k8\" (UniqueName: \"kubernetes.io/projected/b2b49ba9-1957-47ae-8106-88ce782a61cc-kube-api-access-g74k8\") pod \"b2b49ba9-1957-47ae-8106-88ce782a61cc\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") "
May 20 15:58:41.991721 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.991616    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-home\") pod \"b2b49ba9-1957-47ae-8106-88ce782a61cc\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") "
May 20 15:58:41.991721 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.991659    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-model-cache\") pod \"b2b49ba9-1957-47ae-8106-88ce782a61cc\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") "
May 20 15:58:41.991721 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.991717    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-kserve-provision-location\") pod \"b2b49ba9-1957-47ae-8106-88ce782a61cc\" (UID: \"b2b49ba9-1957-47ae-8106-88ce782a61cc\") "
May 20 15:58:41.992109 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.991985    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2b49ba9-1957-47ae-8106-88ce782a61cc" (UID: "b2b49ba9-1957-47ae-8106-88ce782a61cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:58:41.992109 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.992082    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:58:41.992561 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.992273    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-home" (OuterVolumeSpecName: "home") pod "b2b49ba9-1957-47ae-8106-88ce782a61cc" (UID: "b2b49ba9-1957-47ae-8106-88ce782a61cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:58:41.993838 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.993808    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b49ba9-1957-47ae-8106-88ce782a61cc-kube-api-access-g74k8" (OuterVolumeSpecName: "kube-api-access-g74k8") pod "b2b49ba9-1957-47ae-8106-88ce782a61cc" (UID: "b2b49ba9-1957-47ae-8106-88ce782a61cc"). InnerVolumeSpecName "kube-api-access-g74k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:58:41.994012 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.993994    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-dshm" (OuterVolumeSpecName: "dshm") pod "b2b49ba9-1957-47ae-8106-88ce782a61cc" (UID: "b2b49ba9-1957-47ae-8106-88ce782a61cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:58:41.994121 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:41.994101    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b49ba9-1957-47ae-8106-88ce782a61cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2b49ba9-1957-47ae-8106-88ce782a61cc" (UID: "b2b49ba9-1957-47ae-8106-88ce782a61cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:58:42.003739 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.003714    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2b49ba9-1957-47ae-8106-88ce782a61cc" (UID: "b2b49ba9-1957-47ae-8106-88ce782a61cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:58:42.054054 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.054016    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2b49ba9-1957-47ae-8106-88ce782a61cc" (UID: "b2b49ba9-1957-47ae-8106-88ce782a61cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 15:58:42.092870 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.092837    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:58:42.092870 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.092868    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2b49ba9-1957-47ae-8106-88ce782a61cc-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:58:42.093024 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.092878    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:58:42.093024 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.092888    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g74k8\" (UniqueName: \"kubernetes.io/projected/b2b49ba9-1957-47ae-8106-88ce782a61cc-kube-api-access-g74k8\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:58:42.093024 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.092898    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:58:42.093024 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.092907    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2b49ba9-1957-47ae-8106-88ce782a61cc-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:58:42.774463 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.774437    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-58cd7f6c48-cwc2f_b2b49ba9-1957-47ae-8106-88ce782a61cc/main/0.log"
May 20 15:58:42.774842 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.774812    2567 generic.go:358] "Generic (PLEG): container finished" podID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerID="708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151" exitCode=137
May 20 15:58:42.774899 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.774882    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"
May 20 15:58:42.774935 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.774899    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" event={"ID":"b2b49ba9-1957-47ae-8106-88ce782a61cc","Type":"ContainerDied","Data":"708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151"}
May 20 15:58:42.774969 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.774939    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f" event={"ID":"b2b49ba9-1957-47ae-8106-88ce782a61cc","Type":"ContainerDied","Data":"5bc7dfbc7fd798bc3403b5f522b6d031d1dc2cb7ab5b4ec66895b965931a43b9"}
May 20 15:58:42.775000 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.774956    2567 scope.go:117] "RemoveContainer" containerID="708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151"
May 20 15:58:42.785357 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.785336    2567 scope.go:117] "RemoveContainer" containerID="444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab"
May 20 15:58:42.800528 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.800502    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"]
May 20 15:58:42.802570 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.802555    2567 scope.go:117] "RemoveContainer" containerID="708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151"
May 20 15:58:42.802869 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:58:42.802849    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151\": container with ID starting with 708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151 not found: ID does not exist" containerID="708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151"
May 20 15:58:42.802919 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.802878    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151"} err="failed to get container status \"708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151\": rpc error: code = NotFound desc = could not find container \"708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151\": container with ID starting with 708d409a5b52fb05f96f1e46bc3a8e3cf7913a2830b6aca91bb3a00c2bebc151 not found: ID does not exist"
May 20 15:58:42.802919 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.802903    2567 scope.go:117] "RemoveContainer" containerID="444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab"
May 20 15:58:42.803209 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:58:42.803184    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab\": container with ID starting with 444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab not found: ID does not exist" containerID="444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab"
May 20 15:58:42.803297 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.803215    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab"} err="failed to get container status \"444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab\": rpc error: code = NotFound desc = could not find container \"444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab\": container with ID starting with 444a413fe59886d1d31b338d82641a8aec230e4b635e41d72bddf8501ab94fab not found: ID does not exist"
May 20 15:58:42.806678 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:42.806654    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-58cd7f6c48-cwc2f"]
May 20 15:58:43.058742 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:43.058663    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" path="/var/lib/kubelet/pods/b2b49ba9-1957-47ae-8106-88ce782a61cc/volumes"
May 20 15:58:47.360965 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.360920    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-8677dc7484-bsvzf"]
May 20 15:58:47.361543 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.361229    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf" podUID="f85282ae-679b-472b-b395-88f43b596eb0" containerName="manager" containerID="cri-o://c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f" gracePeriod=30
May 20 15:58:47.625190 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.625163    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:58:47.742075 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.742044    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kql7v\" (UniqueName: \"kubernetes.io/projected/f85282ae-679b-472b-b395-88f43b596eb0-kube-api-access-kql7v\") pod \"f85282ae-679b-472b-b395-88f43b596eb0\" (UID: \"f85282ae-679b-472b-b395-88f43b596eb0\") "
May 20 15:58:47.742280 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.742111    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f85282ae-679b-472b-b395-88f43b596eb0-cert\") pod \"f85282ae-679b-472b-b395-88f43b596eb0\" (UID: \"f85282ae-679b-472b-b395-88f43b596eb0\") "
May 20 15:58:47.744246 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.744218    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f85282ae-679b-472b-b395-88f43b596eb0-cert" (OuterVolumeSpecName: "cert") pod "f85282ae-679b-472b-b395-88f43b596eb0" (UID: "f85282ae-679b-472b-b395-88f43b596eb0"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 15:58:47.744360 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.744325    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85282ae-679b-472b-b395-88f43b596eb0-kube-api-access-kql7v" (OuterVolumeSpecName: "kube-api-access-kql7v") pod "f85282ae-679b-472b-b395-88f43b596eb0" (UID: "f85282ae-679b-472b-b395-88f43b596eb0"). InnerVolumeSpecName "kube-api-access-kql7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 15:58:47.797820 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.797782    2567 generic.go:358] "Generic (PLEG): container finished" podID="f85282ae-679b-472b-b395-88f43b596eb0" containerID="c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f" exitCode=0
May 20 15:58:47.797959 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.797849    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf"
May 20 15:58:47.797959 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.797865    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf" event={"ID":"f85282ae-679b-472b-b395-88f43b596eb0","Type":"ContainerDied","Data":"c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f"}
May 20 15:58:47.797959 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.797908    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-8677dc7484-bsvzf" event={"ID":"f85282ae-679b-472b-b395-88f43b596eb0","Type":"ContainerDied","Data":"c175825487f39fb3a780c53510a0398d8628f1e96af4131bb2e895715b018640"}
May 20 15:58:47.797959 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.797928    2567 scope.go:117] "RemoveContainer" containerID="c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f"
May 20 15:58:47.807650 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.807630    2567 scope.go:117] "RemoveContainer" containerID="c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f"
May 20 15:58:47.807918 ip-10-0-130-227 kubenswrapper[2567]: E0520 15:58:47.807897    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f\": container with ID starting with c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f not found: ID does not exist" containerID="c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f"
May 20 15:58:47.807988 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.807925    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f"} err="failed to get container status \"c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f\": rpc error: code = NotFound desc = could not find container \"c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f\": container with ID starting with c070e5f763bc6ca7b8ec5056ce8c2700365ba198a1b9d175fdbe45e1ea1a465f not found: ID does not exist"
May 20 15:58:47.819269 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.819235    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-8677dc7484-bsvzf"]
May 20 15:58:47.823111 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.823089    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-8677dc7484-bsvzf"]
May 20 15:58:47.842920 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.842893    2567 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f85282ae-679b-472b-b395-88f43b596eb0-cert\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:58:47.842920 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:47.842918    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kql7v\" (UniqueName: \"kubernetes.io/projected/f85282ae-679b-472b-b395-88f43b596eb0-kube-api-access-kql7v\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 15:58:49.058309 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:58:49.058278    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f85282ae-679b-472b-b395-88f43b596eb0" path="/var/lib/kubelet/pods/f85282ae-679b-472b-b395-88f43b596eb0/volumes"
May 20 15:59:25.821897 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.821862    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"]
May 20 15:59:25.822365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822262    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main"
May 20 15:59:25.822365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822275    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main"
May 20 15:59:25.822365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822288    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="storage-initializer"
May 20 15:59:25.822365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822293    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="storage-initializer"
May 20 15:59:25.822365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822305    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="storage-initializer"
May 20 15:59:25.822365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822310    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="storage-initializer"
May 20 15:59:25.822365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822322    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main"
May 20 15:59:25.822365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822328    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main"
May 20 15:59:25.822365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822334    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f85282ae-679b-472b-b395-88f43b596eb0" containerName="manager"
May 20 15:59:25.822365 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822341    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f85282ae-679b-472b-b395-88f43b596eb0" containerName="manager"
May 20 15:59:25.822809 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822396    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="155f3e29-d932-410b-8805-77552785cf77" containerName="main"
May 20 15:59:25.822809 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822405    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="b2b49ba9-1957-47ae-8106-88ce782a61cc" containerName="main"
May 20 15:59:25.822809 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.822413    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="f85282ae-679b-472b-b395-88f43b596eb0" containerName="manager"
May 20 15:59:25.825006 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.824987    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.827973 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.827952    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 20 15:59:25.828086 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.827991    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 20 15:59:25.830122 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.830053    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-qg9mj\""
May 20 15:59:25.830314 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.830087    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 20 15:59:25.830633 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.830141    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-sxlvb\""
May 20 15:59:25.838302 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.838274    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"]
May 20 15:59:25.849091 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.849054    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"]
May 20 15:59:25.852033 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.852005    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.872854 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.872816    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"]
May 20 15:59:25.897065 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897031    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.897065 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897069    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.897289 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897088    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grjt\" (UniqueName: \"kubernetes.io/projected/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kube-api-access-4grjt\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.897289 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897168    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.897289 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897196    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.897289 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.897289 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897264    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.897289 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897291    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.897612 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897339    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.897612 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897363    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-dshm\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.897612 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2124c9d-6cc7-43d5-8011-4172cf311064-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.897612 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.897612 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897510    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvfc9\" (UniqueName: \"kubernetes.io/projected/c2124c9d-6cc7-43d5-8011-4172cf311064-kube-api-access-jvfc9\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.897612 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.897535    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-home\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.998294 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998255    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.998294 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998297    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-dshm\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.998537 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998315    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2124c9d-6cc7-43d5-8011-4172cf311064-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.998537 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998358    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.998537 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998379    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvfc9\" (UniqueName: \"kubernetes.io/projected/c2124c9d-6cc7-43d5-8011-4172cf311064-kube-api-access-jvfc9\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.998537 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998399    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-home\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.998537 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998444    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.998838 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998752    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.998838 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998780    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.998838 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998791    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.998838 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998784    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.998838 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998830    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4grjt\" (UniqueName: \"kubernetes.io/projected/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kube-api-access-4grjt\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.998838 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998832    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-home\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.999127 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998902    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.999127 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998943    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.999127 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.998963    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.999127 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.999004    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.999127 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.999031    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.999127 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.999063    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:25.999423 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.999270    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.999491 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.999421    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:25.999549 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:25.999521    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:26.000992 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.000965    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-dshm\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:26.001294 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.001264    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2124c9d-6cc7-43d5-8011-4172cf311064-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:26.001294 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.001284    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:26.001615 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.001575    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:26.009622 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.009567    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grjt\" (UniqueName: \"kubernetes.io/projected/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kube-api-access-4grjt\") pod \"custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:26.011266 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.011246    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvfc9\" (UniqueName: \"kubernetes.io/projected/c2124c9d-6cc7-43d5-8011-4172cf311064-kube-api-access-jvfc9\") pod \"custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:26.136501 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.136458    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:26.164625 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.164578    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 15:59:26.303269 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.303236    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"]
May 20 15:59:26.306114 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:59:26.306084    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2124c9d_6cc7_43d5_8011_4172cf311064.slice/crio-b630ce4cf37ab6c372b90a48d10c45c16cb6d46a60796cc17b44777cfbf76b1c WatchSource:0}: Error finding container b630ce4cf37ab6c372b90a48d10c45c16cb6d46a60796cc17b44777cfbf76b1c: Status 404 returned error can't find the container with id b630ce4cf37ab6c372b90a48d10c45c16cb6d46a60796cc17b44777cfbf76b1c
May 20 15:59:26.330085 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.330055    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"]
May 20 15:59:26.330462 ip-10-0-130-227 kubenswrapper[2567]: W0520 15:59:26.330427    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fc73ff_5e3a_4d0f_bcba_b583e6c73eeb.slice/crio-f4d99577e28942f372134581b506336663caa9e5dd028e7180caf73bba7c7c9f WatchSource:0}: Error finding container f4d99577e28942f372134581b506336663caa9e5dd028e7180caf73bba7c7c9f: Status 404 returned error can't find the container with id f4d99577e28942f372134581b506336663caa9e5dd028e7180caf73bba7c7c9f
May 20 15:59:26.964173 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.964071    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" event={"ID":"c2124c9d-6cc7-43d5-8011-4172cf311064","Type":"ContainerStarted","Data":"b630ce4cf37ab6c372b90a48d10c45c16cb6d46a60796cc17b44777cfbf76b1c"}
May 20 15:59:26.978297 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.977378    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" event={"ID":"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb","Type":"ContainerStarted","Data":"ad07d447194393571defb4beacb74ffa3520257bd1b7006848d3ec13a271cb65"}
May 20 15:59:26.978297 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:26.977425    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" event={"ID":"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb","Type":"ContainerStarted","Data":"f4d99577e28942f372134581b506336663caa9e5dd028e7180caf73bba7c7c9f"}
May 20 15:59:27.986154 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:27.986116    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" event={"ID":"c2124c9d-6cc7-43d5-8011-4172cf311064","Type":"ContainerStarted","Data":"f1e6df667ec2e8d97117c38e333df1823eb676cbec778039818e6668a5bbbab4"}
May 20 15:59:27.987350 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:27.987329    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:28.992571 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:28.992532    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" event={"ID":"c2124c9d-6cc7-43d5-8011-4172cf311064","Type":"ContainerStarted","Data":"ffad5b708e94973f5d6cabfa6278419b3eb68599234a09e2528537d4f6ce3e28"}
May 20 15:59:40.009978 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:40.009944    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 15:59:56.107818 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:56.107770    2567 generic.go:358] "Generic (PLEG): container finished" podID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerID="ad07d447194393571defb4beacb74ffa3520257bd1b7006848d3ec13a271cb65" exitCode=0
May 20 15:59:56.108215 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:56.107840    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" event={"ID":"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb","Type":"ContainerDied","Data":"ad07d447194393571defb4beacb74ffa3520257bd1b7006848d3ec13a271cb65"}
May 20 15:59:57.113338 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:57.113294    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" event={"ID":"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb","Type":"ContainerStarted","Data":"a1c557047f88d8195410787f2ca52798937b6944b18115246624135d0677900d"}
May 20 15:59:57.138135 ip-10-0-130-227 kubenswrapper[2567]: I0520 15:59:57.138072    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" podStartSLOduration=32.138041431 podStartE2EDuration="32.138041431s" podCreationTimestamp="2026-05-20 15:59:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 15:59:57.135533871 +0000 UTC m=+1686.839627473" watchObservedRunningTime="2026-05-20 15:59:57.138041431 +0000 UTC m=+1686.842135020"
May 20 16:00:06.165549 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:00:06.165467    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 16:00:06.165549 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:00:06.165516    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 16:00:06.167186 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:00:06.167152    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
May 20 16:00:16.165603 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:00:16.165541    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
May 20 16:00:26.165495 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:00:26.165448    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
May 20 16:00:36.165863 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:00:36.165823    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
May 20 16:00:46.165663 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:00:46.165622    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
May 20 16:00:56.165233 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:00:56.165181    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
May 20 16:01:06.165767 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:06.165720    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
May 20 16:01:16.165881 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:16.165835    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
May 20 16:01:22.168184 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.168143    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"]
May 20 16:01:22.171849 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.171815    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.174883 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.174858    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 20 16:01:22.185958 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.185931    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"]
May 20 16:01:22.260041 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.260006    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.260231 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.260049    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.260231 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.260085    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.260231 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.260108    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzsh6\" (UniqueName: \"kubernetes.io/projected/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kube-api-access-hzsh6\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.260231 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.260164    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.260231 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.260197    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.260504 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.260230    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.361506 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.361463    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.361703 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.361550    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.361703 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.361633    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.361781 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.361733    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.361824 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.361779    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.361866 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.361820    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.362057 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.361931    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsh6\" (UniqueName: \"kubernetes.io/projected/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kube-api-access-hzsh6\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.362057 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.361946    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.362057 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.362023    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.362342 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.362137    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.362342 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.362304    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.364683 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.364654    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.364832 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.364815    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.371311 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.371279    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzsh6\" (UniqueName: \"kubernetes.io/projected/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kube-api-access-hzsh6\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.482444 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.482353    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:22.635633 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.635601    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"]
May 20 16:01:22.637694 ip-10-0-130-227 kubenswrapper[2567]: W0520 16:01:22.637663    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3a695a_d618_46e7_92c5_7f73c86a39f2.slice/crio-51b4770c8c5180a3541bc4064e756eec22fa326d7ce17a0b3de2f26f5c703a52 WatchSource:0}: Error finding container 51b4770c8c5180a3541bc4064e756eec22fa326d7ce17a0b3de2f26f5c703a52: Status 404 returned error can't find the container with id 51b4770c8c5180a3541bc4064e756eec22fa326d7ce17a0b3de2f26f5c703a52
May 20 16:01:22.639780 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:22.639759    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:01:23.488613 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:23.488558    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" event={"ID":"9f3a695a-d618-46e7-92c5-7f73c86a39f2","Type":"ContainerStarted","Data":"5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5"}
May 20 16:01:23.488613 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:23.488613    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" event={"ID":"9f3a695a-d618-46e7-92c5-7f73c86a39f2","Type":"ContainerStarted","Data":"51b4770c8c5180a3541bc4064e756eec22fa326d7ce17a0b3de2f26f5c703a52"}
May 20 16:01:26.165320 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:26.165272    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
May 20 16:01:27.514879 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:27.514771    2567 generic.go:358] "Generic (PLEG): container finished" podID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerID="5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5" exitCode=0
May 20 16:01:27.514879 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:27.514857    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" event={"ID":"9f3a695a-d618-46e7-92c5-7f73c86a39f2","Type":"ContainerDied","Data":"5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5"}
May 20 16:01:28.521327 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:28.521283    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" event={"ID":"9f3a695a-d618-46e7-92c5-7f73c86a39f2","Type":"ContainerStarted","Data":"2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76"}
May 20 16:01:28.545194 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:28.545130    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podStartSLOduration=6.545107755 podStartE2EDuration="6.545107755s" podCreationTimestamp="2026-05-20 16:01:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:01:28.543710749 +0000 UTC m=+1778.247804340" watchObservedRunningTime="2026-05-20 16:01:28.545107755 +0000 UTC m=+1778.249201346"
May 20 16:01:32.482797 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:32.482701    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:32.482797 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:32.482751    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:01:32.484415 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:32.484384    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
May 20 16:01:36.175178 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:36.175145    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 16:01:36.183181 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:36.183149    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 16:01:42.483686 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:42.483635    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
May 20 16:01:51.143157 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:51.143125    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 16:01:51.149550 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:51.149529    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 16:01:52.483512 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:01:52.483464    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
May 20 16:02:02.483070 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:02:02.483029    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
May 20 16:02:12.483261 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:02:12.483209    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
May 20 16:02:22.483443 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:02:22.483403    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
May 20 16:02:32.483093 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:02:32.483036    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
May 20 16:02:42.483755 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:02:42.483708    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
May 20 16:02:52.483297 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:02:52.483247    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
May 20 16:03:02.483299 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:02.483177    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
May 20 16:03:12.493262 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:12.493233    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:03:12.501250 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:12.501220    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:03:19.074516 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:19.074467    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"]
May 20 16:03:19.075008 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:19.074778    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main" containerID="cri-o://2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76" gracePeriod=30
May 20 16:03:27.864100 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:27.864066    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"]
May 20 16:03:27.867287 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:27.867260    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:27.870823 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:27.870795    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 20 16:03:27.879125 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:27.879095    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"]
May 20 16:03:28.039139 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.039105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-home\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.039346 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.039194    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-dshm\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.039346 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.039225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-model-cache\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.039346 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.039246    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-tmp-dir\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.039346 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.039288    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80c5899-0918-4cb6-9af4-13b26e562464-tls-certs\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.039346 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.039311    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.039346 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.039333    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9srj\" (UniqueName: \"kubernetes.io/projected/e80c5899-0918-4cb6-9af4-13b26e562464-kube-api-access-n9srj\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.140985 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.140887    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-model-cache\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.140985 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.140932    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-tmp-dir\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.141221 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.141169    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80c5899-0918-4cb6-9af4-13b26e562464-tls-certs\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.141287 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.141213    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.141343 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.141320    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n9srj\" (UniqueName: \"kubernetes.io/projected/e80c5899-0918-4cb6-9af4-13b26e562464-kube-api-access-n9srj\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.141418 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.141384    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-model-cache\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.141478 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.141465    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.141682 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.141603    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-home\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.142027 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.142001    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-tmp-dir\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.142131 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.142009    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-dshm\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.142663 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.142641    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-home\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.145000 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.144965    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-dshm\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.145217 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.145190    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80c5899-0918-4cb6-9af4-13b26e562464-tls-certs\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.158816 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.158793    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9srj\" (UniqueName: \"kubernetes.io/projected/e80c5899-0918-4cb6-9af4-13b26e562464-kube-api-access-n9srj\") pod \"custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.179536 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.179502    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:28.327276 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:28.327243    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"]
May 20 16:03:28.328875 ip-10-0-130-227 kubenswrapper[2567]: W0520 16:03:28.328845    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80c5899_0918_4cb6_9af4_13b26e562464.slice/crio-8216ffa4254ea47af1d58c2c04d7576ab68b82e5f9f1ee7e981c7876b7865e84 WatchSource:0}: Error finding container 8216ffa4254ea47af1d58c2c04d7576ab68b82e5f9f1ee7e981c7876b7865e84: Status 404 returned error can't find the container with id 8216ffa4254ea47af1d58c2c04d7576ab68b82e5f9f1ee7e981c7876b7865e84
May 20 16:03:29.031779 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:29.031724    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" event={"ID":"e80c5899-0918-4cb6-9af4-13b26e562464","Type":"ContainerStarted","Data":"f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b"}
May 20 16:03:29.031779 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:29.031779    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" event={"ID":"e80c5899-0918-4cb6-9af4-13b26e562464","Type":"ContainerStarted","Data":"8216ffa4254ea47af1d58c2c04d7576ab68b82e5f9f1ee7e981c7876b7865e84"}
May 20 16:03:36.063870 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:36.063834    2567 generic.go:358] "Generic (PLEG): container finished" podID="e80c5899-0918-4cb6-9af4-13b26e562464" containerID="f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b" exitCode=0
May 20 16:03:36.064525 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:36.063909    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" event={"ID":"e80c5899-0918-4cb6-9af4-13b26e562464","Type":"ContainerDied","Data":"f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b"}
May 20 16:03:37.070346 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:37.070302    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" event={"ID":"e80c5899-0918-4cb6-9af4-13b26e562464","Type":"ContainerStarted","Data":"87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191"}
May 20 16:03:37.094346 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:37.094269    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podStartSLOduration=10.094248759 podStartE2EDuration="10.094248759s" podCreationTimestamp="2026-05-20 16:03:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:03:37.092838009 +0000 UTC m=+1906.796931599" watchObservedRunningTime="2026-05-20 16:03:37.094248759 +0000 UTC m=+1906.798342351"
May 20 16:03:38.180124 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:38.180086    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:38.180520 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:38.180140    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:03:38.181710 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:38.181681    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
May 20 16:03:48.180181 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:48.180130    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
May 20 16:03:49.511123 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.511097    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw_9f3a695a-d618-46e7-92c5-7f73c86a39f2/main/0.log"
May 20 16:03:49.511523 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.511507    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:03:49.563328 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.563293    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-model-cache\") pod \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") "
May 20 16:03:49.563511 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.563339    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tmp-dir\") pod \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") "
May 20 16:03:49.563511 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.563386    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-home\") pod \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") "
May 20 16:03:49.563511 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.563410    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tls-certs\") pod \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") "
May 20 16:03:49.563511 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.563454    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzsh6\" (UniqueName: \"kubernetes.io/projected/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kube-api-access-hzsh6\") pod \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") "
May 20 16:03:49.563511 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.563506    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kserve-provision-location\") pod \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") "
May 20 16:03:49.563838 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.563629    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-dshm\") pod \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\" (UID: \"9f3a695a-d618-46e7-92c5-7f73c86a39f2\") "
May 20 16:03:49.563838 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.563705    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f3a695a-d618-46e7-92c5-7f73c86a39f2" (UID: "9f3a695a-d618-46e7-92c5-7f73c86a39f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:03:49.566032 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.564006    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:03:49.566032 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.564552    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-home" (OuterVolumeSpecName: "home") pod "9f3a695a-d618-46e7-92c5-7f73c86a39f2" (UID: "9f3a695a-d618-46e7-92c5-7f73c86a39f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:03:49.566537 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.566489    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-dshm" (OuterVolumeSpecName: "dshm") pod "9f3a695a-d618-46e7-92c5-7f73c86a39f2" (UID: "9f3a695a-d618-46e7-92c5-7f73c86a39f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:03:49.566686 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.566602    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kube-api-access-hzsh6" (OuterVolumeSpecName: "kube-api-access-hzsh6") pod "9f3a695a-d618-46e7-92c5-7f73c86a39f2" (UID: "9f3a695a-d618-46e7-92c5-7f73c86a39f2"). InnerVolumeSpecName "kube-api-access-hzsh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:03:49.566756 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.566702    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f3a695a-d618-46e7-92c5-7f73c86a39f2" (UID: "9f3a695a-d618-46e7-92c5-7f73c86a39f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:03:49.578530 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.578494    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f3a695a-d618-46e7-92c5-7f73c86a39f2" (UID: "9f3a695a-d618-46e7-92c5-7f73c86a39f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:03:49.604453 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.604373    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f3a695a-d618-46e7-92c5-7f73c86a39f2" (UID: "9f3a695a-d618-46e7-92c5-7f73c86a39f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:03:49.664972 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.664935    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:03:49.664972 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.664971    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:03:49.664972 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.664985    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f3a695a-d618-46e7-92c5-7f73c86a39f2-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:03:49.665229 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.665000    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzsh6\" (UniqueName: \"kubernetes.io/projected/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kube-api-access-hzsh6\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:03:49.665229 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.665014    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:03:49.665229 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:49.665026    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f3a695a-d618-46e7-92c5-7f73c86a39f2-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:03:50.127067 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.127032    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw_9f3a695a-d618-46e7-92c5-7f73c86a39f2/main/0.log"
May 20 16:03:50.127438 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.127412    2567 generic.go:358] "Generic (PLEG): container finished" podID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerID="2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76" exitCode=137
May 20 16:03:50.127515 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.127495    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" event={"ID":"9f3a695a-d618-46e7-92c5-7f73c86a39f2","Type":"ContainerDied","Data":"2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76"}
May 20 16:03:50.127558 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.127503    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"
May 20 16:03:50.127558 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.127538    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw" event={"ID":"9f3a695a-d618-46e7-92c5-7f73c86a39f2","Type":"ContainerDied","Data":"51b4770c8c5180a3541bc4064e756eec22fa326d7ce17a0b3de2f26f5c703a52"}
May 20 16:03:50.127655 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.127558    2567 scope.go:117] "RemoveContainer" containerID="2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76"
May 20 16:03:50.139134 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.139115    2567 scope.go:117] "RemoveContainer" containerID="5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5"
May 20 16:03:50.152004 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.151976    2567 scope.go:117] "RemoveContainer" containerID="2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76"
May 20 16:03:50.152382 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:03:50.152337    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76\": container with ID starting with 2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76 not found: ID does not exist" containerID="2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76"
May 20 16:03:50.152503 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.152393    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76"} err="failed to get container status \"2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76\": rpc error: code = NotFound desc = could not find container \"2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76\": container with ID starting with 2565f4ec4a27390bb8d806bce8b9ece7219dd374a67b4a2b475a55616986ca76 not found: ID does not exist"
May 20 16:03:50.152503 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.152458    2567 scope.go:117] "RemoveContainer" containerID="5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5"
May 20 16:03:50.152930 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:03:50.152845    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5\": container with ID starting with 5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5 not found: ID does not exist" containerID="5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5"
May 20 16:03:50.152930 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.152888    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5"} err="failed to get container status \"5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5\": rpc error: code = NotFound desc = could not find container \"5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5\": container with ID starting with 5610e6559132daa4dfe679ecb47752ed7cbc07d940cec7154a01bcf4b01d0ee5 not found: ID does not exist"
May 20 16:03:50.154175 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.154147    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"]
May 20 16:03:50.158453 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:50.158427    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-6b686f9844xc6pw"]
May 20 16:03:51.058889 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:51.058857    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" path="/var/lib/kubelet/pods/9f3a695a-d618-46e7-92c5-7f73c86a39f2/volumes"
May 20 16:03:58.180166 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:03:58.180120    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
May 20 16:04:08.180729 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:04:08.180674    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
May 20 16:04:18.179954 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:04:18.179901    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
May 20 16:04:28.180764 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:04:28.180726    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
May 20 16:04:38.180985 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:04:38.180886    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
May 20 16:04:48.180888 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:04:48.180840    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
May 20 16:04:58.180564 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:04:58.180517    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
May 20 16:05:08.180983 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:08.180932    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
May 20 16:05:18.190040 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:18.190009    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:05:18.198268 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:18.198233    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:05:24.669035 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:24.668997    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"]
May 20 16:05:24.669996 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:24.669958    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main" containerID="cri-o://87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191" gracePeriod=30
May 20 16:05:34.539261 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.539212    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"]
May 20 16:05:34.539927 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.539906    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="storage-initializer"
May 20 16:05:34.540018 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.539929    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="storage-initializer"
May 20 16:05:34.540018 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.539942    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main"
May 20 16:05:34.540018 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.539950    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main"
May 20 16:05:34.540176 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.540093    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="9f3a695a-d618-46e7-92c5-7f73c86a39f2" containerName="main"
May 20 16:05:34.542471 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.542448    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.545254 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.545231    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 20 16:05:34.553025 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.552994    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"]
May 20 16:05:34.638153 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.638118    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-kserve-provision-location\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.638153 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.638157    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-model-cache\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.638384 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.638188    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw6zn\" (UniqueName: \"kubernetes.io/projected/a2100512-3234-4330-af8d-14fe1cf4ae7c-kube-api-access-mw6zn\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.638384 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.638236    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-home\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.638384 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.638257    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-tmp-dir\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.638384 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.638282    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-dshm\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.638384 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.638358    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2100512-3234-4330-af8d-14fe1cf4ae7c-tls-certs\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.739006 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.738963    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2100512-3234-4330-af8d-14fe1cf4ae7c-tls-certs\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.739210 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.739034    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-kserve-provision-location\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.739210 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.739068    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-model-cache\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.739210 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.739109    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mw6zn\" (UniqueName: \"kubernetes.io/projected/a2100512-3234-4330-af8d-14fe1cf4ae7c-kube-api-access-mw6zn\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.739210 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.739162    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-home\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.739210 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.739187    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-tmp-dir\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.739461 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.739231    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-dshm\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.739693 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.739642    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-home\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.739693 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.739674    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-model-cache\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.739906 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.739712    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-kserve-provision-location\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.739906 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.739753    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-tmp-dir\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.741796 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.741771    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-dshm\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.742251 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.742224    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2100512-3234-4330-af8d-14fe1cf4ae7c-tls-certs\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.748534 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.748511    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw6zn\" (UniqueName: \"kubernetes.io/projected/a2100512-3234-4330-af8d-14fe1cf4ae7c-kube-api-access-mw6zn\") pod \"router-with-refs-test-kserve-8656cc4568-2h6dh\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.853565 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.853530    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:34.999215 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:34.999181    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"]
May 20 16:05:35.001322 ip-10-0-130-227 kubenswrapper[2567]: W0520 16:05:35.001286    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2100512_3234_4330_af8d_14fe1cf4ae7c.slice/crio-d6b52e6ef71deb10d4b9d8b7c20c93e4597ab6082da9deb93fc2b12349af7007 WatchSource:0}: Error finding container d6b52e6ef71deb10d4b9d8b7c20c93e4597ab6082da9deb93fc2b12349af7007: Status 404 returned error can't find the container with id d6b52e6ef71deb10d4b9d8b7c20c93e4597ab6082da9deb93fc2b12349af7007
May 20 16:05:35.576041 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:35.576003    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" event={"ID":"a2100512-3234-4330-af8d-14fe1cf4ae7c","Type":"ContainerStarted","Data":"b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90"}
May 20 16:05:35.576041 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:35.576045    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" event={"ID":"a2100512-3234-4330-af8d-14fe1cf4ae7c","Type":"ContainerStarted","Data":"d6b52e6ef71deb10d4b9d8b7c20c93e4597ab6082da9deb93fc2b12349af7007"}
May 20 16:05:39.599607 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:39.599549    2567 generic.go:358] "Generic (PLEG): container finished" podID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerID="b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90" exitCode=0
May 20 16:05:39.600090 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:39.599629    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" event={"ID":"a2100512-3234-4330-af8d-14fe1cf4ae7c","Type":"ContainerDied","Data":"b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90"}
May 20 16:05:40.605892 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:40.605858    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" event={"ID":"a2100512-3234-4330-af8d-14fe1cf4ae7c","Type":"ContainerStarted","Data":"7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca"}
May 20 16:05:40.632162 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:40.632095    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podStartSLOduration=6.632072277 podStartE2EDuration="6.632072277s" podCreationTimestamp="2026-05-20 16:05:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:05:40.62705938 +0000 UTC m=+2030.331152969" watchObservedRunningTime="2026-05-20 16:05:40.632072277 +0000 UTC m=+2030.336165870"
May 20 16:05:44.854726 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:44.854678    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:44.855150 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:44.854741    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:05:44.856368 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:44.856332    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
May 20 16:05:54.854264 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:54.854217    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
May 20 16:05:55.083676 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.083651    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d_e80c5899-0918-4cb6-9af4-13b26e562464/main/0.log"
May 20 16:05:55.084052 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.084034    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:05:55.249736 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.249644    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9srj\" (UniqueName: \"kubernetes.io/projected/e80c5899-0918-4cb6-9af4-13b26e562464-kube-api-access-n9srj\") pod \"e80c5899-0918-4cb6-9af4-13b26e562464\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") "
May 20 16:05:55.249736 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.249698    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-home\") pod \"e80c5899-0918-4cb6-9af4-13b26e562464\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") "
May 20 16:05:55.249963 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.249757    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-tmp-dir\") pod \"e80c5899-0918-4cb6-9af4-13b26e562464\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") "
May 20 16:05:55.249963 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.249785    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80c5899-0918-4cb6-9af4-13b26e562464-tls-certs\") pod \"e80c5899-0918-4cb6-9af4-13b26e562464\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") "
May 20 16:05:55.249963 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.249825    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-kserve-provision-location\") pod \"e80c5899-0918-4cb6-9af4-13b26e562464\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") "
May 20 16:05:55.249963 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.249863    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-model-cache\") pod \"e80c5899-0918-4cb6-9af4-13b26e562464\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") "
May 20 16:05:55.249963 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.249885    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-dshm\") pod \"e80c5899-0918-4cb6-9af4-13b26e562464\" (UID: \"e80c5899-0918-4cb6-9af4-13b26e562464\") "
May 20 16:05:55.250471 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.250439    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-model-cache" (OuterVolumeSpecName: "model-cache") pod "e80c5899-0918-4cb6-9af4-13b26e562464" (UID: "e80c5899-0918-4cb6-9af4-13b26e562464"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:55.250471 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.250453    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-home" (OuterVolumeSpecName: "home") pod "e80c5899-0918-4cb6-9af4-13b26e562464" (UID: "e80c5899-0918-4cb6-9af4-13b26e562464"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:55.252364 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.252249    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80c5899-0918-4cb6-9af4-13b26e562464-kube-api-access-n9srj" (OuterVolumeSpecName: "kube-api-access-n9srj") pod "e80c5899-0918-4cb6-9af4-13b26e562464" (UID: "e80c5899-0918-4cb6-9af4-13b26e562464"). InnerVolumeSpecName "kube-api-access-n9srj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:05:55.252364 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.252312    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80c5899-0918-4cb6-9af4-13b26e562464-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e80c5899-0918-4cb6-9af4-13b26e562464" (UID: "e80c5899-0918-4cb6-9af4-13b26e562464"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:05:55.252632 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.252616    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-dshm" (OuterVolumeSpecName: "dshm") pod "e80c5899-0918-4cb6-9af4-13b26e562464" (UID: "e80c5899-0918-4cb6-9af4-13b26e562464"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:55.268959 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.268924    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e80c5899-0918-4cb6-9af4-13b26e562464" (UID: "e80c5899-0918-4cb6-9af4-13b26e562464"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:55.290274 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.290230    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e80c5899-0918-4cb6-9af4-13b26e562464" (UID: "e80c5899-0918-4cb6-9af4-13b26e562464"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:05:55.351057 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.351023    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:05:55.351057 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.351053    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e80c5899-0918-4cb6-9af4-13b26e562464-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:05:55.351057 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.351064    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:05:55.351288 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.351073    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:05:55.351288 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.351082    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:05:55.351288 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.351091    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9srj\" (UniqueName: \"kubernetes.io/projected/e80c5899-0918-4cb6-9af4-13b26e562464-kube-api-access-n9srj\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:05:55.351288 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.351099    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e80c5899-0918-4cb6-9af4-13b26e562464-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:05:55.677479 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.677453    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d_e80c5899-0918-4cb6-9af4-13b26e562464/main/0.log"
May 20 16:05:55.677852 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.677824    2567 generic.go:358] "Generic (PLEG): container finished" podID="e80c5899-0918-4cb6-9af4-13b26e562464" containerID="87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191" exitCode=137
May 20 16:05:55.677930 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.677910    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"
May 20 16:05:55.677930 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.677912    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" event={"ID":"e80c5899-0918-4cb6-9af4-13b26e562464","Type":"ContainerDied","Data":"87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191"}
May 20 16:05:55.677999 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.677951    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d" event={"ID":"e80c5899-0918-4cb6-9af4-13b26e562464","Type":"ContainerDied","Data":"8216ffa4254ea47af1d58c2c04d7576ab68b82e5f9f1ee7e981c7876b7865e84"}
May 20 16:05:55.677999 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.677972    2567 scope.go:117] "RemoveContainer" containerID="87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191"
May 20 16:05:55.688318 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.688295    2567 scope.go:117] "RemoveContainer" containerID="f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b"
May 20 16:05:55.703045 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.703017    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"]
May 20 16:05:55.704163 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.704141    2567 scope.go:117] "RemoveContainer" containerID="87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191"
May 20 16:05:55.704513 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:05:55.704485    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191\": container with ID starting with 87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191 not found: ID does not exist" containerID="87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191"
May 20 16:05:55.704652 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.704524    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191"} err="failed to get container status \"87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191\": rpc error: code = NotFound desc = could not find container \"87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191\": container with ID starting with 87d6a998ee849fe1ffa177fd5ad001c08ed2daf0e75f0af98e1627e00f42c191 not found: ID does not exist"
May 20 16:05:55.704652 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.704551    2567 scope.go:117] "RemoveContainer" containerID="f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b"
May 20 16:05:55.704966 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:05:55.704943    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b\": container with ID starting with f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b not found: ID does not exist" containerID="f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b"
May 20 16:05:55.705044 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.704972    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b"} err="failed to get container status \"f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b\": rpc error: code = NotFound desc = could not find container \"f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b\": container with ID starting with f3ade130aa5872470a38855f91d5a77c9589548af726dd56dcc2d78f0179a72b not found: ID does not exist"
May 20 16:05:55.708515 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:55.708490    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-7bb69d5fbd-vqn9d"]
May 20 16:05:57.057533 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:05:57.057495    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" path="/var/lib/kubelet/pods/e80c5899-0918-4cb6-9af4-13b26e562464/volumes"
May 20 16:06:04.854422 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:06:04.854326    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
May 20 16:06:14.854345 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:06:14.854299    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
May 20 16:06:24.854252 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:06:24.854213    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
May 20 16:06:34.854282 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:06:34.854238    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
May 20 16:06:44.854761 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:06:44.854706    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
May 20 16:06:51.210264 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:06:51.210231    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 16:06:51.210785 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:06:51.210234    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 16:06:54.856620 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:06:54.856555    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
May 20 16:07:04.853941 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:04.853896    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
May 20 16:07:14.854744 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:14.854699    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
May 20 16:07:24.863934 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:24.863900    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:07:24.871924 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:24.871896    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:07:31.162610 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:31.162553    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"]
May 20 16:07:31.163520 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:31.163485    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main" containerID="cri-o://7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca" gracePeriod=30
May 20 16:07:41.508194 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.508134    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"]
May 20 16:07:41.508882 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.508858    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main"
May 20 16:07:41.509068 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.509054    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main"
May 20 16:07:41.509211 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.509179    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="storage-initializer"
May 20 16:07:41.509211 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.509198    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="storage-initializer"
May 20 16:07:41.509386 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.509357    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="e80c5899-0918-4cb6-9af4-13b26e562464" containerName="main"
May 20 16:07:41.514574 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.514547    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"]
May 20 16:07:41.514739 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.514727    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.517691 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.517666    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 20 16:07:41.517836 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.517721    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-d2fwf\""
May 20 16:07:41.519470 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.519450    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.529227 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.529195    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"]
May 20 16:07:41.532906 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.532853    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"]
May 20 16:07:41.545014 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.544977    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.545180 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545031    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.545180 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545101    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.545180 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545159    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.545290 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545183    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.545290 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545219    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88gxk\" (UniqueName: \"kubernetes.io/projected/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kube-api-access-88gxk\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.545290 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545249    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.545290 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545273    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.545416 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545312    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.545416 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545390    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.545488 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545433    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.545488 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.545566 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545488    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7bwf\" (UniqueName: \"kubernetes.io/projected/80661f91-2078-43fb-9bd5-73553a309b7d-kube-api-access-q7bwf\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.545566 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.545504    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80661f91-2078-43fb-9bd5-73553a309b7d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.646390 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646353    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-88gxk\" (UniqueName: \"kubernetes.io/projected/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kube-api-access-88gxk\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.646638 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646395    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.646638 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646423    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.646638 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646455    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.646638 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646486    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.646638 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646513    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.646638 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646550    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.646638 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646608    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q7bwf\" (UniqueName: \"kubernetes.io/projected/80661f91-2078-43fb-9bd5-73553a309b7d-kube-api-access-q7bwf\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.646638 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646637    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80661f91-2078-43fb-9bd5-73553a309b7d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.647139 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646682    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.647139 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646709    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.647139 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646737    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.647139 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646790    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.647139 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646822    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.647139 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646940    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.647139 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646959    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.647139 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646982    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.647139 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.646988    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.647139 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.647042    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.647857 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.647207    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.647857 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.647376    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.647857 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.647468    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.649479 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.649454    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.649605 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.649462    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.649872 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.649844    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80661f91-2078-43fb-9bd5-73553a309b7d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.649941 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.649921    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.657281 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.657251    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88gxk\" (UniqueName: \"kubernetes.io/projected/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kube-api-access-88gxk\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.657461 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.657300    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7bwf\" (UniqueName: \"kubernetes.io/projected/80661f91-2078-43fb-9bd5-73553a309b7d-kube-api-access-q7bwf\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:41.831014 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.830795    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:41.837909 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:41.837875    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:42.000437 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:42.000405    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"]
May 20 16:07:42.001213 ip-10-0-130-227 kubenswrapper[2567]: W0520 16:07:42.001187    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80661f91_2078_43fb_9bd5_73553a309b7d.slice/crio-769eac1b46e85b40c98d23f1312bfe59da5d3bf02705d61d0398286d99d0de3b WatchSource:0}: Error finding container 769eac1b46e85b40c98d23f1312bfe59da5d3bf02705d61d0398286d99d0de3b: Status 404 returned error can't find the container with id 769eac1b46e85b40c98d23f1312bfe59da5d3bf02705d61d0398286d99d0de3b
May 20 16:07:42.003189 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:42.003169    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:07:42.026476 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:42.026446    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"]
May 20 16:07:42.027150 ip-10-0-130-227 kubenswrapper[2567]: W0520 16:07:42.027124    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5b5577_c9b4_4c99_949e_7b3203bda7d5.slice/crio-e4d6ca42f4612eb72ef1d29ad6916accdea13d9b542d2328f805cb7a775ffba0 WatchSource:0}: Error finding container e4d6ca42f4612eb72ef1d29ad6916accdea13d9b542d2328f805cb7a775ffba0: Status 404 returned error can't find the container with id e4d6ca42f4612eb72ef1d29ad6916accdea13d9b542d2328f805cb7a775ffba0
May 20 16:07:42.161724 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:42.161684    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" event={"ID":"80661f91-2078-43fb-9bd5-73553a309b7d","Type":"ContainerStarted","Data":"99b2cca7ed9de359b41dff7c79a25320e512f01f5f55a6c3feb303623f959682"}
May 20 16:07:42.161724 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:42.161729    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" event={"ID":"80661f91-2078-43fb-9bd5-73553a309b7d","Type":"ContainerStarted","Data":"769eac1b46e85b40c98d23f1312bfe59da5d3bf02705d61d0398286d99d0de3b"}
May 20 16:07:42.163214 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:42.163182    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" event={"ID":"3f5b5577-c9b4-4c99-949e-7b3203bda7d5","Type":"ContainerStarted","Data":"e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d"}
May 20 16:07:42.163347 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:42.163219    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" event={"ID":"3f5b5577-c9b4-4c99-949e-7b3203bda7d5","Type":"ContainerStarted","Data":"e4d6ca42f4612eb72ef1d29ad6916accdea13d9b542d2328f805cb7a775ffba0"}
May 20 16:07:42.163347 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:42.163244    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:43.170702 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:43.170648    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" event={"ID":"3f5b5577-c9b4-4c99-949e-7b3203bda7d5","Type":"ContainerStarted","Data":"29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3"}
May 20 16:07:47.191780 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:47.191744    2567 generic.go:358] "Generic (PLEG): container finished" podID="80661f91-2078-43fb-9bd5-73553a309b7d" containerID="99b2cca7ed9de359b41dff7c79a25320e512f01f5f55a6c3feb303623f959682" exitCode=0
May 20 16:07:47.192245 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:47.191818    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" event={"ID":"80661f91-2078-43fb-9bd5-73553a309b7d","Type":"ContainerDied","Data":"99b2cca7ed9de359b41dff7c79a25320e512f01f5f55a6c3feb303623f959682"}
May 20 16:07:48.197976 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:48.197936    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" event={"ID":"80661f91-2078-43fb-9bd5-73553a309b7d","Type":"ContainerStarted","Data":"fa343e18ed44bcd4d9dba309f6a1a35f7898074c0ab0d9b99c754860f20ea152"}
May 20 16:07:48.199560 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:48.199533    2567 generic.go:358] "Generic (PLEG): container finished" podID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerID="29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3" exitCode=0
May 20 16:07:48.199712 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:48.199605    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" event={"ID":"3f5b5577-c9b4-4c99-949e-7b3203bda7d5","Type":"ContainerDied","Data":"29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3"}
May 20 16:07:48.219686 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:48.219616    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podStartSLOduration=7.219573947 podStartE2EDuration="7.219573947s" podCreationTimestamp="2026-05-20 16:07:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:07:48.216615713 +0000 UTC m=+2157.920709299" watchObservedRunningTime="2026-05-20 16:07:48.219573947 +0000 UTC m=+2157.923667538"
May 20 16:07:49.207028 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:49.206985    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" event={"ID":"3f5b5577-c9b4-4c99-949e-7b3203bda7d5","Type":"ContainerStarted","Data":"6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9"}
May 20 16:07:49.234661 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:49.234578    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podStartSLOduration=8.234559471 podStartE2EDuration="8.234559471s" podCreationTimestamp="2026-05-20 16:07:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:07:49.231899232 +0000 UTC m=+2158.935992824" watchObservedRunningTime="2026-05-20 16:07:49.234559471 +0000 UTC m=+2158.938653070"
May 20 16:07:51.831560 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:51.831517    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:51.832009 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:51.831676    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:07:51.833226 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:51.833200    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:07:51.838140 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:51.838115    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:51.838296 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:51.838275    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:07:51.839637 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:51.839599    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:07:52.247780 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:07:52.247741    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:08:01.471258 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.471227    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-8656cc4568-2h6dh_a2100512-3234-4330-af8d-14fe1cf4ae7c/main/0.log"
May 20 16:08:01.471731 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.471711    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:08:01.567998 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.567960    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-dshm\") pod \"a2100512-3234-4330-af8d-14fe1cf4ae7c\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") "
May 20 16:08:01.568195 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.568028    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-model-cache\") pod \"a2100512-3234-4330-af8d-14fe1cf4ae7c\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") "
May 20 16:08:01.568195 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.568066    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-kserve-provision-location\") pod \"a2100512-3234-4330-af8d-14fe1cf4ae7c\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") "
May 20 16:08:01.568195 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.568099    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-tmp-dir\") pod \"a2100512-3234-4330-af8d-14fe1cf4ae7c\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") "
May 20 16:08:01.568195 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.568150    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2100512-3234-4330-af8d-14fe1cf4ae7c-tls-certs\") pod \"a2100512-3234-4330-af8d-14fe1cf4ae7c\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") "
May 20 16:08:01.568408 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.568233    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-home\") pod \"a2100512-3234-4330-af8d-14fe1cf4ae7c\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") "
May 20 16:08:01.568408 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.568267    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw6zn\" (UniqueName: \"kubernetes.io/projected/a2100512-3234-4330-af8d-14fe1cf4ae7c-kube-api-access-mw6zn\") pod \"a2100512-3234-4330-af8d-14fe1cf4ae7c\" (UID: \"a2100512-3234-4330-af8d-14fe1cf4ae7c\") "
May 20 16:08:01.568408 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.568359    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2100512-3234-4330-af8d-14fe1cf4ae7c" (UID: "a2100512-3234-4330-af8d-14fe1cf4ae7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:08:01.568704 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.568680    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:08:01.569502 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.569477    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-home" (OuterVolumeSpecName: "home") pod "a2100512-3234-4330-af8d-14fe1cf4ae7c" (UID: "a2100512-3234-4330-af8d-14fe1cf4ae7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:08:01.571017 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.570810    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-dshm" (OuterVolumeSpecName: "dshm") pod "a2100512-3234-4330-af8d-14fe1cf4ae7c" (UID: "a2100512-3234-4330-af8d-14fe1cf4ae7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:08:01.571017 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.571259    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2100512-3234-4330-af8d-14fe1cf4ae7c-kube-api-access-mw6zn" (OuterVolumeSpecName: "kube-api-access-mw6zn") pod "a2100512-3234-4330-af8d-14fe1cf4ae7c" (UID: "a2100512-3234-4330-af8d-14fe1cf4ae7c"). InnerVolumeSpecName "kube-api-access-mw6zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:08:01.571017 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.571355    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2100512-3234-4330-af8d-14fe1cf4ae7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2100512-3234-4330-af8d-14fe1cf4ae7c" (UID: "a2100512-3234-4330-af8d-14fe1cf4ae7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:08:01.591140 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.590945    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2100512-3234-4330-af8d-14fe1cf4ae7c" (UID: "a2100512-3234-4330-af8d-14fe1cf4ae7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:08:01.629437 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.629384    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2100512-3234-4330-af8d-14fe1cf4ae7c" (UID: "a2100512-3234-4330-af8d-14fe1cf4ae7c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:08:01.670260 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.670222    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2100512-3234-4330-af8d-14fe1cf4ae7c-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:08:01.670440 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.670265    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:08:01.670440 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.670286    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw6zn\" (UniqueName: \"kubernetes.io/projected/a2100512-3234-4330-af8d-14fe1cf4ae7c-kube-api-access-mw6zn\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:08:01.670440 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.670303    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:08:01.670440 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.670319    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:08:01.670440 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.670334    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2100512-3234-4330-af8d-14fe1cf4ae7c-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:08:01.831771 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.831662    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:08:01.838686 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:01.838654    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:08:02.283291 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.283250    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-8656cc4568-2h6dh_a2100512-3234-4330-af8d-14fe1cf4ae7c/main/0.log"
May 20 16:08:02.283703 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.283671    2567 generic.go:358] "Generic (PLEG): container finished" podID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerID="7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca" exitCode=137
May 20 16:08:02.283837 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.283723    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" event={"ID":"a2100512-3234-4330-af8d-14fe1cf4ae7c","Type":"ContainerDied","Data":"7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca"}
May 20 16:08:02.283837 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.283768    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"
May 20 16:08:02.283837 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.283778    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh" event={"ID":"a2100512-3234-4330-af8d-14fe1cf4ae7c","Type":"ContainerDied","Data":"d6b52e6ef71deb10d4b9d8b7c20c93e4597ab6082da9deb93fc2b12349af7007"}
May 20 16:08:02.283837 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.283803    2567 scope.go:117] "RemoveContainer" containerID="7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca"
May 20 16:08:02.294360 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.294336    2567 scope.go:117] "RemoveContainer" containerID="b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90"
May 20 16:08:02.306416 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.306390    2567 scope.go:117] "RemoveContainer" containerID="7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca"
May 20 16:08:02.306779 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:08:02.306758    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca\": container with ID starting with 7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca not found: ID does not exist" containerID="7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca"
May 20 16:08:02.306883 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.306787    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca"} err="failed to get container status \"7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca\": rpc error: code = NotFound desc = could not find container \"7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca\": container with ID starting with 7030351728f31f3a59df3dd07f6ffd7c9c067339f5bac9b712afde4e542bf2ca not found: ID does not exist"
May 20 16:08:02.306883 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.306809    2567 scope.go:117] "RemoveContainer" containerID="b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90"
May 20 16:08:02.307083 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:08:02.307064    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90\": container with ID starting with b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90 not found: ID does not exist" containerID="b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90"
May 20 16:08:02.307149 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.307087    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90"} err="failed to get container status \"b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90\": rpc error: code = NotFound desc = could not find container \"b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90\": container with ID starting with b5b51a2ad21558386772a2b30c7f735f372a48ac54f05980f0bd913b327ffc90 not found: ID does not exist"
May 20 16:08:02.313334 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.313300    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"]
May 20 16:08:02.316740 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:02.316707    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-8656cc4568-2h6dh"]
May 20 16:08:03.059426 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:03.059392    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" path="/var/lib/kubelet/pods/a2100512-3234-4330-af8d-14fe1cf4ae7c/volumes"
May 20 16:08:11.831203 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:11.831156    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:08:11.838899 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:11.838851    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:08:21.831994 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:21.831926    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:08:21.838366 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:21.838327    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:08:31.832247 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:31.832194    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:08:31.838651 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:31.838606    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:08:41.831810 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:41.831763    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:08:41.839354 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:41.839314    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:08:51.831983 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:51.831928    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:08:51.838431 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:08:51.838389    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:09:01.832054 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:01.831948    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:09:01.838474 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:01.838431    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:09:11.831467 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:11.831419    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:09:11.839067 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:11.839015    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:09:21.832140 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:21.832078    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:09:21.838704 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:21.838655    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:09:31.831897 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:31.831835    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:09:31.838516 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:31.838477    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:09:41.832066 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:41.832020    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:09:41.838629 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:41.838574    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:09:51.831183 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:51.831131    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:09:51.838938 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:09:51.838904    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:10:01.831640 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:01.831577    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
May 20 16:10:01.839075 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:01.839026    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
May 20 16:10:11.841470 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:11.841423    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:10:11.852296 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:11.852266    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:10:11.855209 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:11.855183    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:10:11.860242 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:11.860214    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:10:20.858855 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:20.858818    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"]
May 20 16:10:20.859421 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:20.859382    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main" containerID="cri-o://6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9" gracePeriod=30
May 20 16:10:20.865763 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:20.865733    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"]
May 20 16:10:20.866168 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:20.866139    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main" containerID="cri-o://fa343e18ed44bcd4d9dba309f6a1a35f7898074c0ab0d9b99c754860f20ea152" gracePeriod=30
May 20 16:10:35.361393 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.361306    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"]
May 20 16:10:35.362026 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.362006    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="storage-initializer"
May 20 16:10:35.362074 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.362030    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="storage-initializer"
May 20 16:10:35.362074 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.362063    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main"
May 20 16:10:35.362074 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.362072    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main"
May 20 16:10:35.362218 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.362206    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="a2100512-3234-4330-af8d-14fe1cf4ae7c" containerName="main"
May 20 16:10:35.367287 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.367255    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.370275 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.370252    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
May 20 16:10:35.375677 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.375646    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"]
May 20 16:10:35.457138 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.457098    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.457344 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.457164    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.457344 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.457253    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkkf\" (UniqueName: \"kubernetes.io/projected/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kube-api-access-lhkkf\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.457344 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.457300    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.457513 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.457398    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.457513 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.457474    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.457644 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.457526    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.558304 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.558248    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.558522 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.558317    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.558522 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.558386    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.558522 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.558432    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.558522 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.558488    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.558780 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.558532    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.558780 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.558687    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkkf\" (UniqueName: \"kubernetes.io/projected/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kube-api-access-lhkkf\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.558780 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.558767    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.558946 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.558797    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.558946 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.558861    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.559043 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.558979    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.561135 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.561111    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.561352 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.561332    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.566897 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.566868    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkkf\" (UniqueName: \"kubernetes.io/projected/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kube-api-access-lhkkf\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.683023 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.682915    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:10:35.831265 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.831233    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"]
May 20 16:10:35.832878 ip-10-0-130-227 kubenswrapper[2567]: W0520 16:10:35.832840    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e3eb9f_0112_489a_bab1_cf2b420ed32f.slice/crio-b95ec387c9d34b1edd01ca2e3f07009ac5027082e083eecd637f5421abce36a7 WatchSource:0}: Error finding container b95ec387c9d34b1edd01ca2e3f07009ac5027082e083eecd637f5421abce36a7: Status 404 returned error can't find the container with id b95ec387c9d34b1edd01ca2e3f07009ac5027082e083eecd637f5421abce36a7
May 20 16:10:35.973647 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.973517    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" event={"ID":"e9e3eb9f-0112-489a-bab1-cf2b420ed32f","Type":"ContainerStarted","Data":"ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226"}
May 20 16:10:35.973647 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:35.973559    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" event={"ID":"e9e3eb9f-0112-489a-bab1-cf2b420ed32f","Type":"ContainerStarted","Data":"b95ec387c9d34b1edd01ca2e3f07009ac5027082e083eecd637f5421abce36a7"}
May 20 16:10:50.859971 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:50.859921    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="llm-d-routing-sidecar" containerID="cri-o://e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d" gracePeriod=2
May 20 16:10:51.047932 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.047903    2567 generic.go:358] "Generic (PLEG): container finished" podID="80661f91-2078-43fb-9bd5-73553a309b7d" containerID="fa343e18ed44bcd4d9dba309f6a1a35f7898074c0ab0d9b99c754860f20ea152" exitCode=137
May 20 16:10:51.048047 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.047981    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" event={"ID":"80661f91-2078-43fb-9bd5-73553a309b7d","Type":"ContainerDied","Data":"fa343e18ed44bcd4d9dba309f6a1a35f7898074c0ab0d9b99c754860f20ea152"}
May 20 16:10:51.049974 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.049945    2567 generic.go:358] "Generic (PLEG): container finished" podID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerID="e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d" exitCode=0
May 20 16:10:51.050067 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.050021    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" event={"ID":"3f5b5577-c9b4-4c99-949e-7b3203bda7d5","Type":"ContainerDied","Data":"e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d"}
May 20 16:10:51.177751 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.177722    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:10:51.322996 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.322964    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80661f91-2078-43fb-9bd5-73553a309b7d-tls-certs\") pod \"80661f91-2078-43fb-9bd5-73553a309b7d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") "
May 20 16:10:51.323203 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.323029    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-tmp-dir\") pod \"80661f91-2078-43fb-9bd5-73553a309b7d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") "
May 20 16:10:51.323203 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.323082    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-home\") pod \"80661f91-2078-43fb-9bd5-73553a309b7d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") "
May 20 16:10:51.323203 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.323102    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-dshm\") pod \"80661f91-2078-43fb-9bd5-73553a309b7d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") "
May 20 16:10:51.323203 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.323142    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-kserve-provision-location\") pod \"80661f91-2078-43fb-9bd5-73553a309b7d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") "
May 20 16:10:51.323203 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.323174    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-model-cache\") pod \"80661f91-2078-43fb-9bd5-73553a309b7d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") "
May 20 16:10:51.323482 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.323213    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7bwf\" (UniqueName: \"kubernetes.io/projected/80661f91-2078-43fb-9bd5-73553a309b7d-kube-api-access-q7bwf\") pod \"80661f91-2078-43fb-9bd5-73553a309b7d\" (UID: \"80661f91-2078-43fb-9bd5-73553a309b7d\") "
May 20 16:10:51.323626 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.323577    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "80661f91-2078-43fb-9bd5-73553a309b7d" (UID: "80661f91-2078-43fb-9bd5-73553a309b7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:51.323866 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.323843    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-home" (OuterVolumeSpecName: "home") pod "80661f91-2078-43fb-9bd5-73553a309b7d" (UID: "80661f91-2078-43fb-9bd5-73553a309b7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:51.325883 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.325843    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80661f91-2078-43fb-9bd5-73553a309b7d-kube-api-access-q7bwf" (OuterVolumeSpecName: "kube-api-access-q7bwf") pod "80661f91-2078-43fb-9bd5-73553a309b7d" (UID: "80661f91-2078-43fb-9bd5-73553a309b7d"). InnerVolumeSpecName "kube-api-access-q7bwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:10:51.325883 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.325864    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-dshm" (OuterVolumeSpecName: "dshm") pod "80661f91-2078-43fb-9bd5-73553a309b7d" (UID: "80661f91-2078-43fb-9bd5-73553a309b7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:51.326013 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.325893    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80661f91-2078-43fb-9bd5-73553a309b7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80661f91-2078-43fb-9bd5-73553a309b7d" (UID: "80661f91-2078-43fb-9bd5-73553a309b7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:10:51.336997 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.336935    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80661f91-2078-43fb-9bd5-73553a309b7d" (UID: "80661f91-2078-43fb-9bd5-73553a309b7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:51.339925 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.339905    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n_3f5b5577-c9b4-4c99-949e-7b3203bda7d5/main/0.log"
May 20 16:10:51.340572 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.340554    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:10:51.384560 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.384513    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "80661f91-2078-43fb-9bd5-73553a309b7d" (UID: "80661f91-2078-43fb-9bd5-73553a309b7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:51.424865 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.424769    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-home\") pod \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") "
May 20 16:10:51.424865 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.424809    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tls-certs\") pod \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") "
May 20 16:10:51.424865 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.424845    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kserve-provision-location\") pod \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") "
May 20 16:10:51.425144 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.424896    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-dshm\") pod \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") "
May 20 16:10:51.425144 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.424933    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tmp-dir\") pod \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") "
May 20 16:10:51.425144 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.424968    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-model-cache\") pod \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") "
May 20 16:10:51.425144 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.425005    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88gxk\" (UniqueName: \"kubernetes.io/projected/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kube-api-access-88gxk\") pod \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\" (UID: \"3f5b5577-c9b4-4c99-949e-7b3203bda7d5\") "
May 20 16:10:51.425355 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.425236    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f5b5577-c9b4-4c99-949e-7b3203bda7d5" (UID: "3f5b5577-c9b4-4c99-949e-7b3203bda7d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:51.425414 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.425384    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.425414 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.425403    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.425518 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.425421    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.425518 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.425435    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.425518 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.425449    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7bwf\" (UniqueName: \"kubernetes.io/projected/80661f91-2078-43fb-9bd5-73553a309b7d-kube-api-access-q7bwf\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.425518 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.425464    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80661f91-2078-43fb-9bd5-73553a309b7d-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.425518 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.425476    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.425518 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.425489    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80661f91-2078-43fb-9bd5-73553a309b7d-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.425518 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.425447    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-home" (OuterVolumeSpecName: "home") pod "3f5b5577-c9b4-4c99-949e-7b3203bda7d5" (UID: "3f5b5577-c9b4-4c99-949e-7b3203bda7d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:51.427389 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.427362    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kube-api-access-88gxk" (OuterVolumeSpecName: "kube-api-access-88gxk") pod "3f5b5577-c9b4-4c99-949e-7b3203bda7d5" (UID: "3f5b5577-c9b4-4c99-949e-7b3203bda7d5"). InnerVolumeSpecName "kube-api-access-88gxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:10:51.427638 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.427608    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f5b5577-c9b4-4c99-949e-7b3203bda7d5" (UID: "3f5b5577-c9b4-4c99-949e-7b3203bda7d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:10:51.427759 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.427664    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-dshm" (OuterVolumeSpecName: "dshm") pod "3f5b5577-c9b4-4c99-949e-7b3203bda7d5" (UID: "3f5b5577-c9b4-4c99-949e-7b3203bda7d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:51.437500 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.437467    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f5b5577-c9b4-4c99-949e-7b3203bda7d5" (UID: "3f5b5577-c9b4-4c99-949e-7b3203bda7d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:51.478288 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.478246    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f5b5577-c9b4-4c99-949e-7b3203bda7d5" (UID: "3f5b5577-c9b4-4c99-949e-7b3203bda7d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:10:51.526956 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.526924    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.526956 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.526953    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.526956 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.526965    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.527190 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.526973    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.527190 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.526983    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:51.527190 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:51.526990    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88gxk\" (UniqueName: \"kubernetes.io/projected/3f5b5577-c9b4-4c99-949e-7b3203bda7d5-kube-api-access-88gxk\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:10:52.056100 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.056071    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"
May 20 16:10:52.056559 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.056071    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d" event={"ID":"80661f91-2078-43fb-9bd5-73553a309b7d","Type":"ContainerDied","Data":"769eac1b46e85b40c98d23f1312bfe59da5d3bf02705d61d0398286d99d0de3b"}
May 20 16:10:52.056559 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.056204    2567 scope.go:117] "RemoveContainer" containerID="fa343e18ed44bcd4d9dba309f6a1a35f7898074c0ab0d9b99c754860f20ea152"
May 20 16:10:52.057699 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.057684    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n_3f5b5577-c9b4-4c99-949e-7b3203bda7d5/main/0.log"
May 20 16:10:52.058453 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.058412    2567 generic.go:358] "Generic (PLEG): container finished" podID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerID="6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9" exitCode=137
May 20 16:10:52.058453 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.058444    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" event={"ID":"3f5b5577-c9b4-4c99-949e-7b3203bda7d5","Type":"ContainerDied","Data":"6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9"}
May 20 16:10:52.058738 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.058477    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n" event={"ID":"3f5b5577-c9b4-4c99-949e-7b3203bda7d5","Type":"ContainerDied","Data":"e4d6ca42f4612eb72ef1d29ad6916accdea13d9b542d2328f805cb7a775ffba0"}
May 20 16:10:52.058738 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.058496    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"
May 20 16:10:52.067635 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.067615    2567 scope.go:117] "RemoveContainer" containerID="99b2cca7ed9de359b41dff7c79a25320e512f01f5f55a6c3feb303623f959682"
May 20 16:10:52.081124 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.081082    2567 scope.go:117] "RemoveContainer" containerID="6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9"
May 20 16:10:52.085130 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.085012    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"]
May 20 16:10:52.090290 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.090259    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-767fp6d"]
May 20 16:10:52.092106 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.092088    2567 scope.go:117] "RemoveContainer" containerID="29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3"
May 20 16:10:52.102084 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.102050    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"]
May 20 16:10:52.106845 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.106817    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7f794dd6855x94n"]
May 20 16:10:52.167506 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.167342    2567 scope.go:117] "RemoveContainer" containerID="e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d"
May 20 16:10:52.180741 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.180716    2567 scope.go:117] "RemoveContainer" containerID="6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9"
May 20 16:10:52.181147 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:10:52.181119    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9\": container with ID starting with 6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9 not found: ID does not exist" containerID="6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9"
May 20 16:10:52.181269 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.181154    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9"} err="failed to get container status \"6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9\": rpc error: code = NotFound desc = could not find container \"6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9\": container with ID starting with 6ff1e0781933f783d06122ed5f2ee9545d541ae62981dbe0ecd8ef1b3a04f0a9 not found: ID does not exist"
May 20 16:10:52.181269 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.181176    2567 scope.go:117] "RemoveContainer" containerID="29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3"
May 20 16:10:52.181500 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:10:52.181482    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3\": container with ID starting with 29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3 not found: ID does not exist" containerID="29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3"
May 20 16:10:52.181565 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.181510    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3"} err="failed to get container status \"29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3\": rpc error: code = NotFound desc = could not find container \"29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3\": container with ID starting with 29d999539b648d183fe94ddaede136b1e5864f9a057c1244e1273f00fa40a0a3 not found: ID does not exist"
May 20 16:10:52.181565 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.181534    2567 scope.go:117] "RemoveContainer" containerID="e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d"
May 20 16:10:52.181849 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:10:52.181825    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d\": container with ID starting with e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d not found: ID does not exist" containerID="e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d"
May 20 16:10:52.181910 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:52.181862    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d"} err="failed to get container status \"e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d\": rpc error: code = NotFound desc = could not find container \"e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d\": container with ID starting with e1e7ed85aacf0108b50b1a6cccbefccad6011d2fd3eee75ea7a304067b2adc8d not found: ID does not exist"
May 20 16:10:53.058060 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:53.058026    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" path="/var/lib/kubelet/pods/3f5b5577-c9b4-4c99-949e-7b3203bda7d5/volumes"
May 20 16:10:53.058618 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:10:53.058601    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" path="/var/lib/kubelet/pods/80661f91-2078-43fb-9bd5-73553a309b7d/volumes"
May 20 16:11:02.100264 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:02.100229    2567 generic.go:358] "Generic (PLEG): container finished" podID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerID="ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226" exitCode=0
May 20 16:11:02.100739 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:02.100303    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" event={"ID":"e9e3eb9f-0112-489a-bab1-cf2b420ed32f","Type":"ContainerDied","Data":"ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226"}
May 20 16:11:03.106090 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:03.106052    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" event={"ID":"e9e3eb9f-0112-489a-bab1-cf2b420ed32f","Type":"ContainerStarted","Data":"7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b"}
May 20 16:11:03.128791 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:03.128731    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podStartSLOduration=28.128710175 podStartE2EDuration="28.128710175s" podCreationTimestamp="2026-05-20 16:10:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:11:03.127891229 +0000 UTC m=+2352.831984818" watchObservedRunningTime="2026-05-20 16:11:03.128710175 +0000 UTC m=+2352.832803764"
May 20 16:11:05.683425 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:05.683393    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:11:05.683425 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:05.683433    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:11:05.685293 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:05.685259    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
May 20 16:11:15.683509 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:15.683457    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
May 20 16:11:25.683807 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:25.683747    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
May 20 16:11:35.684048 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:35.683995    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
May 20 16:11:45.683389 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:45.683342    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
May 20 16:11:51.248618 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:51.248567    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 16:11:51.249873 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:51.249852    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 16:11:55.683391 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:11:55.683341    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
May 20 16:12:05.684241 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:05.684148    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
May 20 16:12:15.683509 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:15.683458    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
May 20 16:12:25.683842 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:25.683797    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
May 20 16:12:35.684338 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:35.684275    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
May 20 16:12:45.693952 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:45.693916    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:12:45.702072 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:45.702040    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:12:50.781352 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:50.781312    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"]
May 20 16:12:50.781791 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:50.781682    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main" containerID="cri-o://7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b" gracePeriod=30
May 20 16:12:57.958655 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.958616    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 16:12:57.959078 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959051    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main"
May 20 16:12:57.959078 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959064    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main"
May 20 16:12:57.959078 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959075    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main"
May 20 16:12:57.959078 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959080    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main"
May 20 16:12:57.959215 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959090    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="llm-d-routing-sidecar"
May 20 16:12:57.959215 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959096    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="llm-d-routing-sidecar"
May 20 16:12:57.959215 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959110    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="storage-initializer"
May 20 16:12:57.959215 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959115    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="storage-initializer"
May 20 16:12:57.959215 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959128    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="storage-initializer"
May 20 16:12:57.959215 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959133    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="storage-initializer"
May 20 16:12:57.959215 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959190    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="llm-d-routing-sidecar"
May 20 16:12:57.959215 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959200    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="80661f91-2078-43fb-9bd5-73553a309b7d" containerName="main"
May 20 16:12:57.959215 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.959210    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="3f5b5577-c9b4-4c99-949e-7b3203bda7d5" containerName="main"
May 20 16:12:57.961649 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.961630    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:57.964392 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.964360    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 20 16:12:57.964544 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.964362    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-k7dhp\""
May 20 16:12:57.975699 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:57.975668    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 16:12:58.090865 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.090815    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.091061 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.090890    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.091061 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.090951    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99c1a40-2233-44fb-8f36-32c7e17d6093-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.091061 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.091013    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.091208 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.091082    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29h9b\" (UniqueName: \"kubernetes.io/projected/a99c1a40-2233-44fb-8f36-32c7e17d6093-kube-api-access-29h9b\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.091208 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.091110    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.091208 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.091126    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.192578 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.192538    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29h9b\" (UniqueName: \"kubernetes.io/projected/a99c1a40-2233-44fb-8f36-32c7e17d6093-kube-api-access-29h9b\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.192578 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.192599    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.192578 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.192621    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.192943 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.192661    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.192943 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.192694    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.192943 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.192735    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99c1a40-2233-44fb-8f36-32c7e17d6093-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.192943 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.192773    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.193211 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.192976    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.193263 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.193226    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.193301 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.193273    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.193339 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.193323    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.195271 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.195245    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.195499 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.195477    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99c1a40-2233-44fb-8f36-32c7e17d6093-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.215450 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.215361    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29h9b\" (UniqueName: \"kubernetes.io/projected/a99c1a40-2233-44fb-8f36-32c7e17d6093-kube-api-access-29h9b\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.273372 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.273324    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:12:58.421352 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.421316    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 16:12:58.423042 ip-10-0-130-227 kubenswrapper[2567]: W0520 16:12:58.423017    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99c1a40_2233_44fb_8f36_32c7e17d6093.slice/crio-7f0e4926f9db26426ec4fc3873a0981bd1a9e381713bd8691a52ed11b93f3f1a WatchSource:0}: Error finding container 7f0e4926f9db26426ec4fc3873a0981bd1a9e381713bd8691a52ed11b93f3f1a: Status 404 returned error can't find the container with id 7f0e4926f9db26426ec4fc3873a0981bd1a9e381713bd8691a52ed11b93f3f1a
May 20 16:12:58.425041 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.425026    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 20 16:12:58.621231 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.621189    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"a99c1a40-2233-44fb-8f36-32c7e17d6093","Type":"ContainerStarted","Data":"0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b"}
May 20 16:12:58.621431 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:12:58.621239    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"a99c1a40-2233-44fb-8f36-32c7e17d6093","Type":"ContainerStarted","Data":"7f0e4926f9db26426ec4fc3873a0981bd1a9e381713bd8691a52ed11b93f3f1a"}
May 20 16:13:03.647836 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:03.647797    2567 generic.go:358] "Generic (PLEG): container finished" podID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerID="0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b" exitCode=0
May 20 16:13:03.649240 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:03.647885    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"a99c1a40-2233-44fb-8f36-32c7e17d6093","Type":"ContainerDied","Data":"0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b"}
May 20 16:13:04.653611 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:04.653554    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"a99c1a40-2233-44fb-8f36-32c7e17d6093","Type":"ContainerStarted","Data":"919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9"}
May 20 16:13:04.673600 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:04.673529    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=7.673513277 podStartE2EDuration="7.673513277s" podCreationTimestamp="2026-05-20 16:12:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:13:04.671940719 +0000 UTC m=+2474.376034311" watchObservedRunningTime="2026-05-20 16:13:04.673513277 +0000 UTC m=+2474.377606903"
May 20 16:13:08.274459 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:08.274418    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:13:08.276159 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:08.276128    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
May 20 16:13:18.274245 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:18.274194    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
May 20 16:13:21.059223 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.059197    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx_e9e3eb9f-0112-489a-bab1-cf2b420ed32f/main/0.log"
May 20 16:13:21.059668 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.059572    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:13:21.228539 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.228502    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhkkf\" (UniqueName: \"kubernetes.io/projected/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kube-api-access-lhkkf\") pod \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") "
May 20 16:13:21.228766 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.228672    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tmp-dir\") pod \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") "
May 20 16:13:21.228766 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.228709    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tls-certs\") pod \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") "
May 20 16:13:21.228766 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.228747    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-dshm\") pod \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") "
May 20 16:13:21.228948 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.228775    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-model-cache\") pod \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") "
May 20 16:13:21.228948 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.228877    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kserve-provision-location\") pod \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") "
May 20 16:13:21.228948 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.228925    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-home\") pod \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\" (UID: \"e9e3eb9f-0112-489a-bab1-cf2b420ed32f\") "
May 20 16:13:21.229097 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.229073    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9e3eb9f-0112-489a-bab1-cf2b420ed32f" (UID: "e9e3eb9f-0112-489a-bab1-cf2b420ed32f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:13:21.229688 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.229657    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:13:21.230122 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.230095    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-home" (OuterVolumeSpecName: "home") pod "e9e3eb9f-0112-489a-bab1-cf2b420ed32f" (UID: "e9e3eb9f-0112-489a-bab1-cf2b420ed32f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:13:21.231292 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.231262    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9e3eb9f-0112-489a-bab1-cf2b420ed32f" (UID: "e9e3eb9f-0112-489a-bab1-cf2b420ed32f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:13:21.231371 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.231320    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-dshm" (OuterVolumeSpecName: "dshm") pod "e9e3eb9f-0112-489a-bab1-cf2b420ed32f" (UID: "e9e3eb9f-0112-489a-bab1-cf2b420ed32f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:13:21.231875 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.231841    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kube-api-access-lhkkf" (OuterVolumeSpecName: "kube-api-access-lhkkf") pod "e9e3eb9f-0112-489a-bab1-cf2b420ed32f" (UID: "e9e3eb9f-0112-489a-bab1-cf2b420ed32f"). InnerVolumeSpecName "kube-api-access-lhkkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:13:21.243371 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.243332    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9e3eb9f-0112-489a-bab1-cf2b420ed32f" (UID: "e9e3eb9f-0112-489a-bab1-cf2b420ed32f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:13:21.309561 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.309449    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9e3eb9f-0112-489a-bab1-cf2b420ed32f" (UID: "e9e3eb9f-0112-489a-bab1-cf2b420ed32f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:13:21.330515 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.330480    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhkkf\" (UniqueName: \"kubernetes.io/projected/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kube-api-access-lhkkf\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:13:21.330515 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.330513    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:13:21.330515 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.330525    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:13:21.330849 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.330533    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:13:21.330849 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.330543    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:13:21.330849 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.330553    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e3eb9f-0112-489a-bab1-cf2b420ed32f-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:13:21.731939 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.731889    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx_e9e3eb9f-0112-489a-bab1-cf2b420ed32f/main/0.log"
May 20 16:13:21.732281 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.732253    2567 generic.go:358] "Generic (PLEG): container finished" podID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerID="7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b" exitCode=137
May 20 16:13:21.732387 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.732320    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"
May 20 16:13:21.732387 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.732337    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" event={"ID":"e9e3eb9f-0112-489a-bab1-cf2b420ed32f","Type":"ContainerDied","Data":"7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b"}
May 20 16:13:21.732387 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.732378    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx" event={"ID":"e9e3eb9f-0112-489a-bab1-cf2b420ed32f","Type":"ContainerDied","Data":"b95ec387c9d34b1edd01ca2e3f07009ac5027082e083eecd637f5421abce36a7"}
May 20 16:13:21.732571 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.732395    2567 scope.go:117] "RemoveContainer" containerID="7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b"
May 20 16:13:21.742482 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.742461    2567 scope.go:117] "RemoveContainer" containerID="ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226"
May 20 16:13:21.753793 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.753770    2567 scope.go:117] "RemoveContainer" containerID="7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b"
May 20 16:13:21.754092 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:13:21.754073    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b\": container with ID starting with 7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b not found: ID does not exist" containerID="7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b"
May 20 16:13:21.754147 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.754101    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b"} err="failed to get container status \"7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b\": rpc error: code = NotFound desc = could not find container \"7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b\": container with ID starting with 7d57516483b3f782afb32a9ab475395e41a43752353da00f96e2324f7a9d776b not found: ID does not exist"
May 20 16:13:21.754147 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.754123    2567 scope.go:117] "RemoveContainer" containerID="ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226"
May 20 16:13:21.754332 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:13:21.754318    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226\": container with ID starting with ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226 not found: ID does not exist" containerID="ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226"
May 20 16:13:21.754368 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.754337    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226"} err="failed to get container status \"ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226\": rpc error: code = NotFound desc = could not find container \"ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226\": container with ID starting with ea9b8271ee8fa2a958abe69e2bfb0d7f0029abe3dc32cd4919906d555523b226 not found: ID does not exist"
May 20 16:13:21.759904 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.759875    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"]
May 20 16:13:21.762552 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:21.762522    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7b85fbf5f6ld6bx"]
May 20 16:13:23.057885 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:23.057842    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" path="/var/lib/kubelet/pods/e9e3eb9f-0112-489a-bab1-cf2b420ed32f/volumes"
May 20 16:13:28.273737 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:28.273700    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:13:28.274306 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:28.273982    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
May 20 16:13:38.273947 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:38.273851    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
May 20 16:13:48.273917 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:48.273874    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
May 20 16:13:58.274633 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:13:58.274562    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
May 20 16:14:08.274497 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:08.274450    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
May 20 16:14:18.274378 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:18.274333    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
May 20 16:14:20.932694 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:20.932652    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"]
May 20 16:14:20.933153 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:20.933031    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main" containerID="cri-o://a1c557047f88d8195410787f2ca52798937b6944b18115246624135d0677900d" gracePeriod=30
May 20 16:14:20.942128 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:20.942093    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"]
May 20 16:14:20.942534 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:20.942484    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="llm-d-routing-sidecar" containerID="cri-o://f1e6df667ec2e8d97117c38e333df1823eb676cbec778039818e6668a5bbbab4" gracePeriod=30
May 20 16:14:20.942683 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:20.942529    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="storage-initializer" containerID="cri-o://ffad5b708e94973f5d6cabfa6278419b3eb68599234a09e2528537d4f6ce3e28" gracePeriod=30
May 20 16:14:22.000451 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:22.000418    2567 generic.go:358] "Generic (PLEG): container finished" podID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerID="f1e6df667ec2e8d97117c38e333df1823eb676cbec778039818e6668a5bbbab4" exitCode=0
May 20 16:14:22.000868 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:22.000469    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" event={"ID":"c2124c9d-6cc7-43d5-8011-4172cf311064","Type":"ContainerDied","Data":"f1e6df667ec2e8d97117c38e333df1823eb676cbec778039818e6668a5bbbab4"}
May 20 16:14:26.137376 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:26.137318    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
May 20 16:14:28.274681 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:28.274634    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
May 20 16:14:29.997522 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:29.997477    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
May 20 16:14:36.137866 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:36.137817    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
May 20 16:14:38.273876 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:38.273828    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
May 20 16:14:39.997959 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:39.997915    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
May 20 16:14:46.137678 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:46.137630    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
May 20 16:14:46.138074 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:46.137721    2567 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 16:14:48.284314 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:48.284283    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:14:48.291880 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:48.291838    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:14:49.997801 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:49.997758    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
May 20 16:14:51.129336 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.129288    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb_c2124c9d-6cc7-43d5-8011-4172cf311064/storage-initializer/0.log"
May 20 16:14:51.130044 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.129673    2567 generic.go:358] "Generic (PLEG): container finished" podID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerID="ffad5b708e94973f5d6cabfa6278419b3eb68599234a09e2528537d4f6ce3e28" exitCode=137
May 20 16:14:51.130044 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.129830    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" event={"ID":"c2124c9d-6cc7-43d5-8011-4172cf311064","Type":"ContainerDied","Data":"ffad5b708e94973f5d6cabfa6278419b3eb68599234a09e2528537d4f6ce3e28"}
May 20 16:14:51.132071 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.132021    2567 generic.go:358] "Generic (PLEG): container finished" podID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerID="a1c557047f88d8195410787f2ca52798937b6944b18115246624135d0677900d" exitCode=137
May 20 16:14:51.132171 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.132101    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" event={"ID":"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb","Type":"ContainerDied","Data":"a1c557047f88d8195410787f2ca52798937b6944b18115246624135d0677900d"}
May 20 16:14:51.179314 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.179283    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb_c2124c9d-6cc7-43d5-8011-4172cf311064/storage-initializer/0.log"
May 20 16:14:51.179808 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.179787    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 16:14:51.221917 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.221893    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 16:14:51.223811 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.223788    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-model-cache\") pod \"c2124c9d-6cc7-43d5-8011-4172cf311064\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") "
May 20 16:14:51.223946 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.223865    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-tmp-dir\") pod \"c2124c9d-6cc7-43d5-8011-4172cf311064\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") "
May 20 16:14:51.223946 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.223921    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvfc9\" (UniqueName: \"kubernetes.io/projected/c2124c9d-6cc7-43d5-8011-4172cf311064-kube-api-access-jvfc9\") pod \"c2124c9d-6cc7-43d5-8011-4172cf311064\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") "
May 20 16:14:51.224021 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.223963    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2124c9d-6cc7-43d5-8011-4172cf311064-tls-certs\") pod \"c2124c9d-6cc7-43d5-8011-4172cf311064\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") "
May 20 16:14:51.224021 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.224002    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-dshm\") pod \"c2124c9d-6cc7-43d5-8011-4172cf311064\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") "
May 20 16:14:51.224110 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.224041    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-home\") pod \"c2124c9d-6cc7-43d5-8011-4172cf311064\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") "
May 20 16:14:51.224110 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.224075    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2124c9d-6cc7-43d5-8011-4172cf311064" (UID: "c2124c9d-6cc7-43d5-8011-4172cf311064"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:51.224212 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.224109    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-kserve-provision-location\") pod \"c2124c9d-6cc7-43d5-8011-4172cf311064\" (UID: \"c2124c9d-6cc7-43d5-8011-4172cf311064\") "
May 20 16:14:51.224212 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.224155    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2124c9d-6cc7-43d5-8011-4172cf311064" (UID: "c2124c9d-6cc7-43d5-8011-4172cf311064"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:51.224364 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.224317    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-home" (OuterVolumeSpecName: "home") pod "c2124c9d-6cc7-43d5-8011-4172cf311064" (UID: "c2124c9d-6cc7-43d5-8011-4172cf311064"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:51.224502 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.224438    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.224502 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.224456    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.224502 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.224470    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.226385 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.226356    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2124c9d-6cc7-43d5-8011-4172cf311064-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2124c9d-6cc7-43d5-8011-4172cf311064" (UID: "c2124c9d-6cc7-43d5-8011-4172cf311064"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:14:51.226661 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.226641    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-dshm" (OuterVolumeSpecName: "dshm") pod "c2124c9d-6cc7-43d5-8011-4172cf311064" (UID: "c2124c9d-6cc7-43d5-8011-4172cf311064"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:51.226789 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.226692    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2124c9d-6cc7-43d5-8011-4172cf311064-kube-api-access-jvfc9" (OuterVolumeSpecName: "kube-api-access-jvfc9") pod "c2124c9d-6cc7-43d5-8011-4172cf311064" (UID: "c2124c9d-6cc7-43d5-8011-4172cf311064"). InnerVolumeSpecName "kube-api-access-jvfc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:14:51.273212 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.273166    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2124c9d-6cc7-43d5-8011-4172cf311064" (UID: "c2124c9d-6cc7-43d5-8011-4172cf311064"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:51.325144 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.325109    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-model-cache\") pod \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") "
May 20 16:14:51.325334 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.325168    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tls-certs\") pod \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") "
May 20 16:14:51.325334 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.325193    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tmp-dir\") pod \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") "
May 20 16:14:51.325334 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.325259    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4grjt\" (UniqueName: \"kubernetes.io/projected/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kube-api-access-4grjt\") pod \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") "
May 20 16:14:51.325334 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.325284    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-dshm\") pod \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") "
May 20 16:14:51.325569 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.325414    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kserve-provision-location\") pod \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") "
May 20 16:14:51.325569 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.325508    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-home\") pod \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\" (UID: \"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb\") "
May 20 16:14:51.326349 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.325901    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.326349 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.325930    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvfc9\" (UniqueName: \"kubernetes.io/projected/c2124c9d-6cc7-43d5-8011-4172cf311064-kube-api-access-jvfc9\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.326349 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.325946    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2124c9d-6cc7-43d5-8011-4172cf311064-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.326349 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.325961    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2124c9d-6cc7-43d5-8011-4172cf311064-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.326349 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.326223    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" (UID: "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:51.327108 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.327067    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-home" (OuterVolumeSpecName: "home") pod "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" (UID: "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:51.327876 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.327844    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-dshm" (OuterVolumeSpecName: "dshm") pod "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" (UID: "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:51.328063 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.328041    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" (UID: "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:14:51.328130 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.328040    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kube-api-access-4grjt" (OuterVolumeSpecName: "kube-api-access-4grjt") pod "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" (UID: "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb"). InnerVolumeSpecName "kube-api-access-4grjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:14:51.336378 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.336351    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" (UID: "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:51.345810 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.345778    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" (UID: "a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:14:51.427350 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.427317    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.427350 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.427347    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.427563 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.427360    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4grjt\" (UniqueName: \"kubernetes.io/projected/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kube-api-access-4grjt\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.427563 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.427370    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.427563 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.427379    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.427563 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.427388    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:51.427563 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:51.427397    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:14:52.137659 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.137636    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb_c2124c9d-6cc7-43d5-8011-4172cf311064/storage-initializer/0.log"
May 20 16:14:52.138109 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.138092    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"
May 20 16:14:52.138159 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.138116    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb" event={"ID":"c2124c9d-6cc7-43d5-8011-4172cf311064","Type":"ContainerDied","Data":"b630ce4cf37ab6c372b90a48d10c45c16cb6d46a60796cc17b44777cfbf76b1c"}
May 20 16:14:52.138193 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.138167    2567 scope.go:117] "RemoveContainer" containerID="ffad5b708e94973f5d6cabfa6278419b3eb68599234a09e2528537d4f6ce3e28"
May 20 16:14:52.140071 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.140049    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"
May 20 16:14:52.140180 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.140047    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt" event={"ID":"a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb","Type":"ContainerDied","Data":"f4d99577e28942f372134581b506336663caa9e5dd028e7180caf73bba7c7c9f"}
May 20 16:14:52.171383 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.171359    2567 scope.go:117] "RemoveContainer" containerID="f1e6df667ec2e8d97117c38e333df1823eb676cbec778039818e6668a5bbbab4"
May 20 16:14:52.181686 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.181657    2567 scope.go:117] "RemoveContainer" containerID="a1c557047f88d8195410787f2ca52798937b6944b18115246624135d0677900d"
May 20 16:14:52.190093 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.190054    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"]
May 20 16:14:52.192967 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.192943    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6869f69dfd-f5fgb"]
May 20 16:14:52.193932 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.193905    2567 scope.go:117] "RemoveContainer" containerID="ad07d447194393571defb4beacb74ffa3520257bd1b7006848d3ec13a271cb65"
May 20 16:14:52.204306 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.204277    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"]
May 20 16:14:52.206827 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.206801    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-67b5d66757-d85jt"]
May 20 16:14:52.784875 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.784840    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"]
May 20 16:14:52.785272 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785258    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="storage-initializer"
May 20 16:14:52.785319 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785274    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="storage-initializer"
May 20 16:14:52.785319 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785293    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="storage-initializer"
May 20 16:14:52.785319 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785299    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="storage-initializer"
May 20 16:14:52.785319 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785305    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main"
May 20 16:14:52.785319 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785311    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main"
May 20 16:14:52.785319 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785318    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="storage-initializer"
May 20 16:14:52.785501 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785323    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="storage-initializer"
May 20 16:14:52.785501 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785332    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main"
May 20 16:14:52.785501 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785338    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main"
May 20 16:14:52.785501 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785350    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="llm-d-routing-sidecar"
May 20 16:14:52.785501 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785355    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="llm-d-routing-sidecar"
May 20 16:14:52.785501 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785411    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9e3eb9f-0112-489a-bab1-cf2b420ed32f" containerName="main"
May 20 16:14:52.785501 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785421    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="storage-initializer"
May 20 16:14:52.785501 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785429    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" containerName="main"
May 20 16:14:52.785501 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.785435    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" containerName="llm-d-routing-sidecar"
May 20 16:14:52.791980 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.791954    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.794816 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.794792    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 20 16:14:52.795121 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.795101    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-5cdcm\""
May 20 16:14:52.798576 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.798552    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"]
May 20 16:14:52.841377 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.841316    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-home\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.841377 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.841365    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-model-cache\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.841629 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.841458    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.841629 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.841531    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-dshm\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.841724 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.841650    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tls-certs\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.841724 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.841667    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tmp-dir\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.841724 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.841686    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg6fk\" (UniqueName: \"kubernetes.io/projected/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kube-api-access-kg6fk\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.942311 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.942266    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tls-certs\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.942311 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.942309    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tmp-dir\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.942565 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.942327    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kg6fk\" (UniqueName: \"kubernetes.io/projected/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kube-api-access-kg6fk\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.942565 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.942363    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-home\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.942565 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.942383    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-model-cache\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.942565 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.942408    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.942565 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.942455    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-dshm\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.942873 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.942797    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tmp-dir\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.942873 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.942808    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-home\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.942978 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.942887    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.942978 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.942934    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-model-cache\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.944919 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.944899    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-dshm\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.945095 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.945080    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tls-certs\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:52.949498 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:52.949479    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg6fk\" (UniqueName: \"kubernetes.io/projected/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kube-api-access-kg6fk\") pod \"router-with-refs-pd-test-kserve-54bb9d745d-vsxjp\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:53.058258 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:53.058165    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb" path="/var/lib/kubelet/pods/a2fc73ff-5e3a-4d0f-bcba-b583e6c73eeb/volumes"
May 20 16:14:53.058824 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:53.058799    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2124c9d-6cc7-43d5-8011-4172cf311064" path="/var/lib/kubelet/pods/c2124c9d-6cc7-43d5-8011-4172cf311064/volumes"
May 20 16:14:53.104122 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:53.104083    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:53.249112 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:53.249083    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"]
May 20 16:14:53.250465 ip-10-0-130-227 kubenswrapper[2567]: W0520 16:14:53.250438    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce98f95_37bc_498e_9ae9_60da7cb4ad22.slice/crio-6f685589fb90a448568b2ca5ba13ea54b101bdfab52b76d956e5694395b68712 WatchSource:0}: Error finding container 6f685589fb90a448568b2ca5ba13ea54b101bdfab52b76d956e5694395b68712: Status 404 returned error can't find the container with id 6f685589fb90a448568b2ca5ba13ea54b101bdfab52b76d956e5694395b68712
May 20 16:14:54.153640 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:54.153519    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" event={"ID":"fce98f95-37bc-498e-9ae9-60da7cb4ad22","Type":"ContainerStarted","Data":"0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9"}
May 20 16:14:54.153640 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:54.153565    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" event={"ID":"fce98f95-37bc-498e-9ae9-60da7cb4ad22","Type":"ContainerStarted","Data":"6f685589fb90a448568b2ca5ba13ea54b101bdfab52b76d956e5694395b68712"}
May 20 16:14:54.153832 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:54.153649    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:14:55.161005 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:55.160966    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" event={"ID":"fce98f95-37bc-498e-9ae9-60da7cb4ad22","Type":"ContainerStarted","Data":"c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35"}
May 20 16:14:59.182145 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:59.182109    2567 generic.go:358] "Generic (PLEG): container finished" podID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerID="c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35" exitCode=0
May 20 16:14:59.182713 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:14:59.182181    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" event={"ID":"fce98f95-37bc-498e-9ae9-60da7cb4ad22","Type":"ContainerDied","Data":"c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35"}
May 20 16:15:00.190195 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:00.190155    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" event={"ID":"fce98f95-37bc-498e-9ae9-60da7cb4ad22","Type":"ContainerStarted","Data":"4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877"}
May 20 16:15:00.213158 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:00.213084    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podStartSLOduration=8.213061431 podStartE2EDuration="8.213061431s" podCreationTimestamp="2026-05-20 16:14:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:15:00.210537327 +0000 UTC m=+2589.914630917" watchObservedRunningTime="2026-05-20 16:15:00.213061431 +0000 UTC m=+2589.917155024"
May 20 16:15:03.104783 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:03.104677    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:15:03.105269 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:03.105198    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:15:03.107138 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:03.107095    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
May 20 16:15:03.123758 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:03.123727    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:15:04.402372 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:04.402335    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 16:15:04.403251 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:04.403216    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main" containerID="cri-o://919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9" gracePeriod=30
May 20 16:15:05.179453 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.179424    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:15:05.221848 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.221757    2567 generic.go:358] "Generic (PLEG): container finished" podID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerID="919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9" exitCode=0
May 20 16:15:05.221848 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.221799    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"a99c1a40-2233-44fb-8f36-32c7e17d6093","Type":"ContainerDied","Data":"919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9"}
May 20 16:15:05.221848 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.221825    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"a99c1a40-2233-44fb-8f36-32c7e17d6093","Type":"ContainerDied","Data":"7f0e4926f9db26426ec4fc3873a0981bd1a9e381713bd8691a52ed11b93f3f1a"}
May 20 16:15:05.221848 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.221840    2567 scope.go:117] "RemoveContainer" containerID="919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9"
May 20 16:15:05.222191 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.221862    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 20 16:15:05.232093 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.232066    2567 scope.go:117] "RemoveContainer" containerID="0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b"
May 20 16:15:05.269078 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.269039    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99c1a40-2233-44fb-8f36-32c7e17d6093-tls-certs\") pod \"a99c1a40-2233-44fb-8f36-32c7e17d6093\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") "
May 20 16:15:05.269078 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.269088    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-model-cache\") pod \"a99c1a40-2233-44fb-8f36-32c7e17d6093\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") "
May 20 16:15:05.269326 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.269147    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-tmp-dir\") pod \"a99c1a40-2233-44fb-8f36-32c7e17d6093\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") "
May 20 16:15:05.269326 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.269169    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-home\") pod \"a99c1a40-2233-44fb-8f36-32c7e17d6093\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") "
May 20 16:15:05.269326 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.269208    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-dshm\") pod \"a99c1a40-2233-44fb-8f36-32c7e17d6093\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") "
May 20 16:15:05.269326 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.269231    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29h9b\" (UniqueName: \"kubernetes.io/projected/a99c1a40-2233-44fb-8f36-32c7e17d6093-kube-api-access-29h9b\") pod \"a99c1a40-2233-44fb-8f36-32c7e17d6093\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") "
May 20 16:15:05.269326 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.269265    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-kserve-provision-location\") pod \"a99c1a40-2233-44fb-8f36-32c7e17d6093\" (UID: \"a99c1a40-2233-44fb-8f36-32c7e17d6093\") "
May 20 16:15:05.269743 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.269700    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-model-cache" (OuterVolumeSpecName: "model-cache") pod "a99c1a40-2233-44fb-8f36-32c7e17d6093" (UID: "a99c1a40-2233-44fb-8f36-32c7e17d6093"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:15:05.269892 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.269867    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-home" (OuterVolumeSpecName: "home") pod "a99c1a40-2233-44fb-8f36-32c7e17d6093" (UID: "a99c1a40-2233-44fb-8f36-32c7e17d6093"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:15:05.272045 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.272012    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99c1a40-2233-44fb-8f36-32c7e17d6093-kube-api-access-29h9b" (OuterVolumeSpecName: "kube-api-access-29h9b") pod "a99c1a40-2233-44fb-8f36-32c7e17d6093" (UID: "a99c1a40-2233-44fb-8f36-32c7e17d6093"). InnerVolumeSpecName "kube-api-access-29h9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:15:05.272174 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.272159    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99c1a40-2233-44fb-8f36-32c7e17d6093-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a99c1a40-2233-44fb-8f36-32c7e17d6093" (UID: "a99c1a40-2233-44fb-8f36-32c7e17d6093"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:15:05.272548 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.272530    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-dshm" (OuterVolumeSpecName: "dshm") pod "a99c1a40-2233-44fb-8f36-32c7e17d6093" (UID: "a99c1a40-2233-44fb-8f36-32c7e17d6093"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:15:05.291204 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.291156    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a99c1a40-2233-44fb-8f36-32c7e17d6093" (UID: "a99c1a40-2233-44fb-8f36-32c7e17d6093"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:15:05.309176 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.309147    2567 scope.go:117] "RemoveContainer" containerID="919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9"
May 20 16:15:05.309576 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:15:05.309540    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9\": container with ID starting with 919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9 not found: ID does not exist" containerID="919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9"
May 20 16:15:05.309657 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.309604    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9"} err="failed to get container status \"919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9\": rpc error: code = NotFound desc = could not find container \"919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9\": container with ID starting with 919287f42cd698eef9b349977d6b41ca4c819ec0fb3b8f9435bedfc8936066a9 not found: ID does not exist"
May 20 16:15:05.309657 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.309633    2567 scope.go:117] "RemoveContainer" containerID="0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b"
May 20 16:15:05.309956 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:15:05.309929    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b\": container with ID starting with 0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b not found: ID does not exist" containerID="0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b"
May 20 16:15:05.310021 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.309966    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b"} err="failed to get container status \"0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b\": rpc error: code = NotFound desc = could not find container \"0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b\": container with ID starting with 0dfb9004863d7068d46f173778122235b1de2da3800f8e561b9cdd843617624b not found: ID does not exist"
May 20 16:15:05.332111 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.332061    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a99c1a40-2233-44fb-8f36-32c7e17d6093" (UID: "a99c1a40-2233-44fb-8f36-32c7e17d6093"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:15:05.370693 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.370657    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a99c1a40-2233-44fb-8f36-32c7e17d6093-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:15:05.370693 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.370685    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:15:05.370693 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.370694    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:15:05.370693 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.370702    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:15:05.370944 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.370710    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:15:05.370944 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.370720    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29h9b\" (UniqueName: \"kubernetes.io/projected/a99c1a40-2233-44fb-8f36-32c7e17d6093-kube-api-access-29h9b\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:15:05.370944 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.370729    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a99c1a40-2233-44fb-8f36-32c7e17d6093-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:15:05.550144 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.550109    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 16:15:05.553012 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:05.552985    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 20 16:15:07.058684 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:07.058648    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" path="/var/lib/kubelet/pods/a99c1a40-2233-44fb-8f36-32c7e17d6093/volumes"
May 20 16:15:13.105342 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:13.105297    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
May 20 16:15:23.105051 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:23.104993    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
May 20 16:15:33.104911 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:33.104857    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
May 20 16:15:43.104529 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:43.104481    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
May 20 16:15:53.104748 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:15:53.104691    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
May 20 16:16:03.105023 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:16:03.104965    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
May 20 16:16:13.104771 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:16:13.104721    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
May 20 16:16:23.105485 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:16:23.105435    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
May 20 16:16:33.105657 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:16:33.105397    2567 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8001/health\": dial tcp 10.134.0.63:8001: connect: connection refused"
May 20 16:16:43.114558 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:16:43.114524    2567 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:16:43.127250 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:16:43.127220    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:16:51.297535 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:16:51.297493    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 16:16:51.303068 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:16:51.303031    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 16:16:54.717490 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:16:54.717448    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"]
May 20 16:16:54.718015 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:16:54.717947    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main" containerID="cri-o://4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877" gracePeriod=30
May 20 16:17:11.720486 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:11.720449    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:11.734812 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:11.734775    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:11.744460 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:11.744432    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:12.729770 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:12.729734    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:12.737663 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:12.737630    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:12.746971 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:12.746941    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:13.737603 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:13.737548    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:13.745083 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:13.745046    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:13.754773 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:13.754732    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:14.720882 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:14.720848    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:14.730659 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:14.730627    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:14.741230 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:14.741192    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:15.726923 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:15.726882    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:15.736320 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:15.736293    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:15.746566 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:15.746536    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:16.728132 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:16.728099    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:16.734905 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:16.734861    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:16.746351 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:16.746321    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:17.723548 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:17.723518    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:17.732090 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:17.732046    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:17.742415 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:17.742365    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:18.715607 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:18.715555    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:18.723566 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:18.723533    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:18.733936 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:18.733905    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:19.703163 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:19.703124    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:19.709853 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:19.709825    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:19.719301 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:19.719271    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:20.678761 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:20.678711    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:20.685458 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:20.685422    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:20.695348 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:20.695318    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:21.700186 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:21.700156    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:21.708214 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:21.708183    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:21.717540 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:21.717482    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:22.754343 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:22.754303    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:22.761019 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:22.760992    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:22.770881 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:22.770852    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:23.802765 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:23.802724    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:23.811936 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:23.811902    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:23.827796 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:23.827760    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:24.718789 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:24.718714    2567 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="llm-d-routing-sidecar" containerID="cri-o://0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9" gracePeriod=2
May 20 16:17:24.866181 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:24.866155    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:24.873559 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:24.873529    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/llm-d-routing-sidecar/0.log"
May 20 16:17:24.883515 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:24.883484    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/storage-initializer/0.log"
May 20 16:17:25.011352 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.011324    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:25.012046 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.012024    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:17:25.128508 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.128467    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-home\") pod \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") "
May 20 16:17:25.128729 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.128538    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-model-cache\") pod \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") "
May 20 16:17:25.128729 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.128570    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tls-certs\") pod \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") "
May 20 16:17:25.128729 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.128687    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tmp-dir\") pod \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") "
May 20 16:17:25.128912 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.128741    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg6fk\" (UniqueName: \"kubernetes.io/projected/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kube-api-access-kg6fk\") pod \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") "
May 20 16:17:25.128912 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.128782    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kserve-provision-location\") pod \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") "
May 20 16:17:25.128912 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.128814    2567 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-dshm\") pod \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\" (UID: \"fce98f95-37bc-498e-9ae9-60da7cb4ad22\") "
May 20 16:17:25.128912 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.128886    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-model-cache" (OuterVolumeSpecName: "model-cache") pod "fce98f95-37bc-498e-9ae9-60da7cb4ad22" (UID: "fce98f95-37bc-498e-9ae9-60da7cb4ad22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:17:25.129141 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.129118    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-model-cache\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:17:25.129628 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.129578    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-home" (OuterVolumeSpecName: "home") pod "fce98f95-37bc-498e-9ae9-60da7cb4ad22" (UID: "fce98f95-37bc-498e-9ae9-60da7cb4ad22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:17:25.131198 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.131171    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-dshm" (OuterVolumeSpecName: "dshm") pod "fce98f95-37bc-498e-9ae9-60da7cb4ad22" (UID: "fce98f95-37bc-498e-9ae9-60da7cb4ad22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:17:25.131198 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.131181    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fce98f95-37bc-498e-9ae9-60da7cb4ad22" (UID: "fce98f95-37bc-498e-9ae9-60da7cb4ad22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 20 16:17:25.131367 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.131290    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kube-api-access-kg6fk" (OuterVolumeSpecName: "kube-api-access-kg6fk") pod "fce98f95-37bc-498e-9ae9-60da7cb4ad22" (UID: "fce98f95-37bc-498e-9ae9-60da7cb4ad22"). InnerVolumeSpecName "kube-api-access-kg6fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 20 16:17:25.142930 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.142891    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fce98f95-37bc-498e-9ae9-60da7cb4ad22" (UID: "fce98f95-37bc-498e-9ae9-60da7cb4ad22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:17:25.196209 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.196156    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fce98f95-37bc-498e-9ae9-60da7cb4ad22" (UID: "fce98f95-37bc-498e-9ae9-60da7cb4ad22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 20 16:17:25.229740 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.229701    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tls-certs\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:17:25.229740 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.229734    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-tmp-dir\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:17:25.229740 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.229745    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg6fk\" (UniqueName: \"kubernetes.io/projected/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kube-api-access-kg6fk\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:17:25.229970 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.229754    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-kserve-provision-location\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:17:25.229970 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.229763    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-dshm\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:17:25.229970 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.229771    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce98f95-37bc-498e-9ae9-60da7cb4ad22-home\") on node \"ip-10-0-130-227.ec2.internal\" DevicePath \"\""
May 20 16:17:25.819475 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.819439    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-54bb9d745d-vsxjp_fce98f95-37bc-498e-9ae9-60da7cb4ad22/main/0.log"
May 20 16:17:25.820252 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.820217    2567 generic.go:358] "Generic (PLEG): container finished" podID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerID="4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877" exitCode=137
May 20 16:17:25.820252 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.820248    2567 generic.go:358] "Generic (PLEG): container finished" podID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerID="0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9" exitCode=0
May 20 16:17:25.820552 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.820293    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" event={"ID":"fce98f95-37bc-498e-9ae9-60da7cb4ad22","Type":"ContainerDied","Data":"4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877"}
May 20 16:17:25.820552 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.820337    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" event={"ID":"fce98f95-37bc-498e-9ae9-60da7cb4ad22","Type":"ContainerDied","Data":"0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9"}
May 20 16:17:25.820552 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.820351    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp" event={"ID":"fce98f95-37bc-498e-9ae9-60da7cb4ad22","Type":"ContainerDied","Data":"6f685589fb90a448568b2ca5ba13ea54b101bdfab52b76d956e5694395b68712"}
May 20 16:17:25.820552 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.820354    2567 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"
May 20 16:17:25.820552 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.820366    2567 scope.go:117] "RemoveContainer" containerID="4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877"
May 20 16:17:25.834351 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.834329    2567 scope.go:117] "RemoveContainer" containerID="c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35"
May 20 16:17:25.849897 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.849855    2567 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"]
May 20 16:17:25.851771 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.851705    2567 scope.go:117] "RemoveContainer" containerID="0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9"
May 20 16:17:25.853473 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.853439    2567 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-54bb9d745d-vsxjp"]
May 20 16:17:25.865449 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.865407    2567 scope.go:117] "RemoveContainer" containerID="4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877"
May 20 16:17:25.865832 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:17:25.865787    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877\": container with ID starting with 4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877 not found: ID does not exist" containerID="4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877"
May 20 16:17:25.865960 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.865835    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877"} err="failed to get container status \"4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877\": rpc error: code = NotFound desc = could not find container \"4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877\": container with ID starting with 4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877 not found: ID does not exist"
May 20 16:17:25.865960 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.865865    2567 scope.go:117] "RemoveContainer" containerID="c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35"
May 20 16:17:25.866206 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:17:25.866174    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35\": container with ID starting with c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35 not found: ID does not exist" containerID="c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35"
May 20 16:17:25.866621 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.866229    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35"} err="failed to get container status \"c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35\": rpc error: code = NotFound desc = could not find container \"c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35\": container with ID starting with c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35 not found: ID does not exist"
May 20 16:17:25.866621 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.866257    2567 scope.go:117] "RemoveContainer" containerID="0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9"
May 20 16:17:25.866773 ip-10-0-130-227 kubenswrapper[2567]: E0520 16:17:25.866672    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9\": container with ID starting with 0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9 not found: ID does not exist" containerID="0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9"
May 20 16:17:25.866773 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.866702    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9"} err="failed to get container status \"0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9\": rpc error: code = NotFound desc = could not find container \"0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9\": container with ID starting with 0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9 not found: ID does not exist"
May 20 16:17:25.866773 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.866727    2567 scope.go:117] "RemoveContainer" containerID="4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877"
May 20 16:17:25.867032 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.867001    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877"} err="failed to get container status \"4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877\": rpc error: code = NotFound desc = could not find container \"4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877\": container with ID starting with 4df286afed22dc0aff7ba67b6c35636291168e6ed4d1d2950d0de3ee47045877 not found: ID does not exist"
May 20 16:17:25.867149 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.867033    2567 scope.go:117] "RemoveContainer" containerID="c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35"
May 20 16:17:25.867347 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.867320    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35"} err="failed to get container status \"c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35\": rpc error: code = NotFound desc = could not find container \"c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35\": container with ID starting with c395f391dc0cdcfb1e00ae3b64dc4140c753ef40a0485b41c748fbee1d21ed35 not found: ID does not exist"
May 20 16:17:25.867427 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.867352    2567 scope.go:117] "RemoveContainer" containerID="0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9"
May 20 16:17:25.867809 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.867781    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9"} err="failed to get container status \"0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9\": rpc error: code = NotFound desc = could not find container \"0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9\": container with ID starting with 0e43e86d5a1e9a7eeee47dcaae0cc8f5b3bd43bd5e297fcd6baa0c42ce3024c9 not found: ID does not exist"
May 20 16:17:25.926321 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.926288    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-pkc4d_d0fc4c44-95c4-4f18-8d90-8345bab17426/discovery/0.log"
May 20 16:17:25.939889 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:25.939846    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-849cdfbf5f-nm88r_82ef32e9-7004-4dc8-bc95-fe1c8413b759/istio-proxy/0.log"
May 20 16:17:26.759376 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:26.759339    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-pkc4d_d0fc4c44-95c4-4f18-8d90-8345bab17426/discovery/0.log"
May 20 16:17:26.772605 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:26.772548    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-849cdfbf5f-nm88r_82ef32e9-7004-4dc8-bc95-fe1c8413b759/istio-proxy/0.log"
May 20 16:17:27.057521 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:27.057427    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" path="/var/lib/kubelet/pods/fce98f95-37bc-498e-9ae9-60da7cb4ad22/volumes"
May 20 16:17:27.545845 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:27.545817    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7587b89b76-h5ll8_b19a1204-3297-4a76-a566-7ba97481abd0/manager/0.log"
May 20 16:17:27.636823 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:27.636793    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-2svzj_622c88fc-eecb-4d0a-b074-aca2f27ca4f0/limitador/0.log"
May 20 16:17:27.652364 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:27.652331    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-bgqqp_f051a663-4c2a-4524-803e-fc68ffdbc8a9/manager/0.log"
May 20 16:17:29.928488 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.928450    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m228h/must-gather-7nf2n"]
May 20 16:17:29.928977 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.928884    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="llm-d-routing-sidecar"
May 20 16:17:29.928977 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.928896    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="llm-d-routing-sidecar"
May 20 16:17:29.928977 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.928910    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="storage-initializer"
May 20 16:17:29.928977 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.928916    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="storage-initializer"
May 20 16:17:29.928977 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.928923    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main"
May 20 16:17:29.928977 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.928930    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main"
May 20 16:17:29.928977 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.928942    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main"
May 20 16:17:29.928977 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.928947    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main"
May 20 16:17:29.928977 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.928964    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="storage-initializer"
May 20 16:17:29.928977 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.928969    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="storage-initializer"
May 20 16:17:29.929329 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.929021    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="llm-d-routing-sidecar"
May 20 16:17:29.929329 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.929032    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="a99c1a40-2233-44fb-8f36-32c7e17d6093" containerName="main"
May 20 16:17:29.929329 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.929038    2567 memory_manager.go:356] "RemoveStaleState removing state" podUID="fce98f95-37bc-498e-9ae9-60da7cb4ad22" containerName="main"
May 20 16:17:29.934795 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.934768    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m228h/must-gather-7nf2n"
May 20 16:17:29.937897 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.937866    2567 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m228h\"/\"default-dockercfg-br4zm\""
May 20 16:17:29.938085 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.937907    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m228h\"/\"kube-root-ca.crt\""
May 20 16:17:29.938085 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.938041    2567 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m228h\"/\"openshift-service-ca.crt\""
May 20 16:17:29.940624 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:29.940599    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m228h/must-gather-7nf2n"]
May 20 16:17:30.075278 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:30.075241    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4rn\" (UniqueName: \"kubernetes.io/projected/ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7-kube-api-access-pc4rn\") pod \"must-gather-7nf2n\" (UID: \"ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7\") " pod="openshift-must-gather-m228h/must-gather-7nf2n"
May 20 16:17:30.075483 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:30.075366    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7-must-gather-output\") pod \"must-gather-7nf2n\" (UID: \"ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7\") " pod="openshift-must-gather-m228h/must-gather-7nf2n"
May 20 16:17:30.176060 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:30.176026    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7-must-gather-output\") pod \"must-gather-7nf2n\" (UID: \"ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7\") " pod="openshift-must-gather-m228h/must-gather-7nf2n"
May 20 16:17:30.176253 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:30.176111    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4rn\" (UniqueName: \"kubernetes.io/projected/ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7-kube-api-access-pc4rn\") pod \"must-gather-7nf2n\" (UID: \"ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7\") " pod="openshift-must-gather-m228h/must-gather-7nf2n"
May 20 16:17:30.176423 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:30.176400    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7-must-gather-output\") pod \"must-gather-7nf2n\" (UID: \"ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7\") " pod="openshift-must-gather-m228h/must-gather-7nf2n"
May 20 16:17:30.184719 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:30.184649    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4rn\" (UniqueName: \"kubernetes.io/projected/ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7-kube-api-access-pc4rn\") pod \"must-gather-7nf2n\" (UID: \"ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7\") " pod="openshift-must-gather-m228h/must-gather-7nf2n"
May 20 16:17:30.252821 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:30.252771    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m228h/must-gather-7nf2n"
May 20 16:17:30.393426 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:30.393388    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m228h/must-gather-7nf2n"]
May 20 16:17:30.394434 ip-10-0-130-227 kubenswrapper[2567]: W0520 16:17:30.394403    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecb09cb7_6a12_4258_aea3_c5ebe95fa5b7.slice/crio-c518f186fecf6e9eb2bdae6a4f01b8d24672d9fed3452a0dfee6b05fc1c7b748 WatchSource:0}: Error finding container c518f186fecf6e9eb2bdae6a4f01b8d24672d9fed3452a0dfee6b05fc1c7b748: Status 404 returned error can't find the container with id c518f186fecf6e9eb2bdae6a4f01b8d24672d9fed3452a0dfee6b05fc1c7b748
May 20 16:17:30.843918 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:30.843867    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m228h/must-gather-7nf2n" event={"ID":"ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7","Type":"ContainerStarted","Data":"c518f186fecf6e9eb2bdae6a4f01b8d24672d9fed3452a0dfee6b05fc1c7b748"}
May 20 16:17:31.849365 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:31.849325    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m228h/must-gather-7nf2n" event={"ID":"ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7","Type":"ContainerStarted","Data":"ef2a4efc05a3991368762511058ad76523d081b315efcb62201b85fa5fa280d4"}
May 20 16:17:31.849365 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:31.849365    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m228h/must-gather-7nf2n" event={"ID":"ecb09cb7-6a12-4258-aea3-c5ebe95fa5b7","Type":"ContainerStarted","Data":"bea19ec8ec264828f52ccf731d77554d298df353e2516137183146ef89c8075f"}
May 20 16:17:31.866070 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:31.865947    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-m228h/must-gather-7nf2n" podStartSLOduration=1.9437113529999999 podStartE2EDuration="2.865930127s" podCreationTimestamp="2026-05-20 16:17:29 +0000 UTC" firstStartedPulling="2026-05-20 16:17:30.396174036 +0000 UTC m=+2740.100267604" lastFinishedPulling="2026-05-20 16:17:31.3183928 +0000 UTC m=+2741.022486378" observedRunningTime="2026-05-20 16:17:31.864356802 +0000 UTC m=+2741.568450392" watchObservedRunningTime="2026-05-20 16:17:31.865930127 +0000 UTC m=+2741.570023717"
May 20 16:17:34.935692 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:34.935651    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j6glz_c2c2975f-9436-4c8c-a470-7a2c0d401942/global-pull-secret-syncer/0.log"
May 20 16:17:34.983498 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:34.983439    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b25vj_71f15969-7e79-4580-b24d-41f479c3715e/konnectivity-agent/0.log"
May 20 16:17:35.079167 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:35.079138    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-227.ec2.internal_a90538a519c49afc5a39f91384815237/haproxy/0.log"
May 20 16:17:39.641624 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:39.641572    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7587b89b76-h5ll8_b19a1204-3297-4a76-a566-7ba97481abd0/manager/0.log"
May 20 16:17:39.780377 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:39.780331    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-2svzj_622c88fc-eecb-4d0a-b074-aca2f27ca4f0/limitador/0.log"
May 20 16:17:39.820034 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:39.819688    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-bgqqp_f051a663-4c2a-4524-803e-fc68ffdbc8a9/manager/0.log"
May 20 16:17:40.771928 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:40.771892    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25db283d-ab2f-4d00-b513-7a50b9481b91/alertmanager/0.log"
May 20 16:17:40.794547 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:40.794512    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25db283d-ab2f-4d00-b513-7a50b9481b91/config-reloader/0.log"
May 20 16:17:40.813370 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:40.813342    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25db283d-ab2f-4d00-b513-7a50b9481b91/kube-rbac-proxy-web/0.log"
May 20 16:17:40.834206 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:40.834173    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25db283d-ab2f-4d00-b513-7a50b9481b91/kube-rbac-proxy/0.log"
May 20 16:17:40.856339 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:40.856296    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25db283d-ab2f-4d00-b513-7a50b9481b91/kube-rbac-proxy-metric/0.log"
May 20 16:17:40.877358 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:40.877322    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25db283d-ab2f-4d00-b513-7a50b9481b91/prom-label-proxy/0.log"
May 20 16:17:40.903379 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:40.903349    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25db283d-ab2f-4d00-b513-7a50b9481b91/init-config-reloader/0.log"
May 20 16:17:40.979877 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:40.979820    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-77qrl_30991255-505c-4f6a-8afd-3d25afc9b842/kube-state-metrics/0.log"
May 20 16:17:41.006281 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.006252    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-77qrl_30991255-505c-4f6a-8afd-3d25afc9b842/kube-rbac-proxy-main/0.log"
May 20 16:17:41.033248 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.033156    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-77qrl_30991255-505c-4f6a-8afd-3d25afc9b842/kube-rbac-proxy-self/0.log"
May 20 16:17:41.061427 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.061391    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6d685b6d6c-pn6kp_a46d6274-fc02-4bea-8ba5-31040d263711/metrics-server/0.log"
May 20 16:17:41.087170 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.087129    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-wvntl_3a204308-a018-4037-a3db-8bc7a0e5444c/monitoring-plugin/0.log"
May 20 16:17:41.280663 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.280626    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cpr9c_5a0e960d-021a-4d9c-921a-7c3f884015d1/node-exporter/0.log"
May 20 16:17:41.303393 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.303282    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cpr9c_5a0e960d-021a-4d9c-921a-7c3f884015d1/kube-rbac-proxy/0.log"
May 20 16:17:41.324087 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.324055    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cpr9c_5a0e960d-021a-4d9c-921a-7c3f884015d1/init-textfile/0.log"
May 20 16:17:41.359276 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.359230    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-r2m4h_507701a3-3d79-4d09-ba31-8e27a16b7edf/kube-rbac-proxy-main/0.log"
May 20 16:17:41.385298 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.385247    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-r2m4h_507701a3-3d79-4d09-ba31-8e27a16b7edf/kube-rbac-proxy-self/0.log"
May 20 16:17:41.407637 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.407606    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-r2m4h_507701a3-3d79-4d09-ba31-8e27a16b7edf/openshift-state-metrics/0.log"
May 20 16:17:41.632170 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.632139    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-l4vwx_c0cc31f6-0dab-44f5-8917-e52473c3da68/prometheus-operator/0.log"
May 20 16:17:41.661518 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.661491    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-l4vwx_c0cc31f6-0dab-44f5-8917-e52473c3da68/kube-rbac-proxy/0.log"
May 20 16:17:41.716397 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.716359    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bc566fb97-8bt9g_1fe0f182-2f5b-41bc-a208-6df929d947da/telemeter-client/0.log"
May 20 16:17:41.739609 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.738867    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bc566fb97-8bt9g_1fe0f182-2f5b-41bc-a208-6df929d947da/reload/0.log"
May 20 16:17:41.769220 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.769181    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6bc566fb97-8bt9g_1fe0f182-2f5b-41bc-a208-6df929d947da/kube-rbac-proxy/0.log"
May 20 16:17:41.804136 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.804099    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d796ddbc9-bbf6k_fc05d954-00b2-4086-a171-f90af0bdaf0f/thanos-query/0.log"
May 20 16:17:41.823622 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.823573    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d796ddbc9-bbf6k_fc05d954-00b2-4086-a171-f90af0bdaf0f/kube-rbac-proxy-web/0.log"
May 20 16:17:41.843925 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.843893    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d796ddbc9-bbf6k_fc05d954-00b2-4086-a171-f90af0bdaf0f/kube-rbac-proxy/0.log"
May 20 16:17:41.867100 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.867068    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d796ddbc9-bbf6k_fc05d954-00b2-4086-a171-f90af0bdaf0f/prom-label-proxy/0.log"
May 20 16:17:41.891279 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.891191    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d796ddbc9-bbf6k_fc05d954-00b2-4086-a171-f90af0bdaf0f/kube-rbac-proxy-rules/0.log"
May 20 16:17:41.922427 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:41.922384    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d796ddbc9-bbf6k_fc05d954-00b2-4086-a171-f90af0bdaf0f/kube-rbac-proxy-metrics/0.log"
May 20 16:17:44.036946 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.036912    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9695b8856-x4klq_5cae9eea-5aeb-4330-9d8f-3e5d6fa20887/console/0.log"
May 20 16:17:44.080902 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.080872    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-xvk6q_0316ba6c-6a22-4aca-af00-0a93ecf7a6e3/download-server/0.log"
May 20 16:17:44.270094 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.270050    2567 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"]
May 20 16:17:44.276360 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.276332    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.282140 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.282106    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"]
May 20 16:17:44.433756 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.433713    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-proc\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.434125 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.434101    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsllt\" (UniqueName: \"kubernetes.io/projected/1171c6e6-0f7d-45de-beb3-fe6447ec665f-kube-api-access-lsllt\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.434299 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.434275    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-lib-modules\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.434421 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.434338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-sys\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.434421 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.434396    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-podres\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.535964 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.535923    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-sys\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.536173 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.535982    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-podres\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.536173 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.536011    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-proc\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.536173 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.536062    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-sys\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.536173 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.536135    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lsllt\" (UniqueName: \"kubernetes.io/projected/1171c6e6-0f7d-45de-beb3-fe6447ec665f-kube-api-access-lsllt\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.536173 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.536144    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-proc\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.536173 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.536160    2567 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-lib-modules\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.536173 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.536161    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-podres\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.536548 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.536297    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1171c6e6-0f7d-45de-beb3-fe6447ec665f-lib-modules\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.547267 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.547232    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsllt\" (UniqueName: \"kubernetes.io/projected/1171c6e6-0f7d-45de-beb3-fe6447ec665f-kube-api-access-lsllt\") pod \"perf-node-gather-daemonset-5rhrw\" (UID: \"1171c6e6-0f7d-45de-beb3-fe6447ec665f\") " pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.591397 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.591354    2567 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.747521 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.747454    2567 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"]
May 20 16:17:44.934882 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.934778    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw" event={"ID":"1171c6e6-0f7d-45de-beb3-fe6447ec665f","Type":"ContainerStarted","Data":"2f4e8772d2a4804b10cc1fae5e51a4c4fef0c35e3f75153d49c296f463434de8"}
May 20 16:17:44.934882 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.934831    2567 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw" event={"ID":"1171c6e6-0f7d-45de-beb3-fe6447ec665f","Type":"ContainerStarted","Data":"9bd33f6765b84784303e1348859ac7bdc33578880a2d1d6692fd08292ceb68a3"}
May 20 16:17:44.935731 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.935702    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:44.955210 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:44.954732    2567 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw" podStartSLOduration=0.954711366 podStartE2EDuration="954.711366ms" podCreationTimestamp="2026-05-20 16:17:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-20 16:17:44.95047087 +0000 UTC m=+2754.654564461" watchObservedRunningTime="2026-05-20 16:17:44.954711366 +0000 UTC m=+2754.658804968"
May 20 16:17:45.309265 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:45.309175    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x555j_67023ceb-0ace-44c4-9111-eec6cfaf0212/dns/0.log"
May 20 16:17:45.328453 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:45.328427    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x555j_67023ceb-0ace-44c4-9111-eec6cfaf0212/kube-rbac-proxy/0.log"
May 20 16:17:45.373764 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:45.373714    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dpk92_0ab595d7-4e45-464c-87e7-58d498cbb575/dns-node-resolver/0.log"
May 20 16:17:45.861905 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:45.861872    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ccd2v_ffa2ed51-616b-4441-a33e-45c761dd075e/node-ca/0.log"
May 20 16:17:46.788449 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:46.788417    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-798958bb55-pkc4d_d0fc4c44-95c4-4f18-8d90-8345bab17426/discovery/0.log"
May 20 16:17:46.810655 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:46.810623    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-849cdfbf5f-nm88r_82ef32e9-7004-4dc8-bc95-fe1c8413b759/istio-proxy/0.log"
May 20 16:17:47.255476 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:47.255433    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2gp62_2a9b92fe-cce1-4050-85b1-1f4b12e178b6/serve-healthcheck-canary/0.log"
May 20 16:17:47.897101 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:47.897067    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p574b_8d481c73-56b1-4d54-97e3-9344490303ee/kube-rbac-proxy/0.log"
May 20 16:17:47.915544 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:47.915510    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p574b_8d481c73-56b1-4d54-97e3-9344490303ee/exporter/0.log"
May 20 16:17:47.936902 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:47.936870    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p574b_8d481c73-56b1-4d54-97e3-9344490303ee/extractor/0.log"
May 20 16:17:50.944496 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:50.944451    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-87b4b9896-p89ct_7e549444-3b41-44b3-bc39-669145afbabd/manager/0.log"
May 20 16:17:51.291151 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:51.291070    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-znl9n_891539dc-9b96-4d97-a03a-fa8826801d9b/seaweedfs/0.log"
May 20 16:17:51.956807 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:51.956780    2567 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m228h/perf-node-gather-daemonset-5rhrw"
May 20 16:17:57.115861 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:57.115814    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4t7b8_6b36f593-5d2c-45dc-b0ff-bd5e90381257/kube-multus/0.log"
May 20 16:17:57.454133 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:57.454051    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsxb5_e367b95c-7b4d-4d9c-8a71-1429e26b0230/kube-multus-additional-cni-plugins/0.log"
May 20 16:17:57.473561 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:57.473532    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsxb5_e367b95c-7b4d-4d9c-8a71-1429e26b0230/egress-router-binary-copy/0.log"
May 20 16:17:57.493029 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:57.492999    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsxb5_e367b95c-7b4d-4d9c-8a71-1429e26b0230/cni-plugins/0.log"
May 20 16:17:57.513802 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:57.513772    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsxb5_e367b95c-7b4d-4d9c-8a71-1429e26b0230/bond-cni-plugin/0.log"
May 20 16:17:57.532024 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:57.531999    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsxb5_e367b95c-7b4d-4d9c-8a71-1429e26b0230/routeoverride-cni/0.log"
May 20 16:17:57.552924 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:57.552891    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsxb5_e367b95c-7b4d-4d9c-8a71-1429e26b0230/whereabouts-cni-bincopy/0.log"
May 20 16:17:57.572337 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:57.572308    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fsxb5_e367b95c-7b4d-4d9c-8a71-1429e26b0230/whereabouts-cni/0.log"
May 20 16:17:57.665839 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:57.665805    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-46dsm_9f8d9d11-6bdb-402e-ab74-7485d6c52a79/network-metrics-daemon/0.log"
May 20 16:17:57.682558 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:57.682529    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-46dsm_9f8d9d11-6bdb-402e-ab74-7485d6c52a79/kube-rbac-proxy/0.log"
May 20 16:17:58.528845 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:58.528804    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-controller/0.log"
May 20 16:17:58.544437 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:58.544406    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/0.log"
May 20 16:17:58.571118 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:58.571084    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovn-acl-logging/1.log"
May 20 16:17:58.591868 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:58.591826    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/kube-rbac-proxy-node/0.log"
May 20 16:17:58.612865 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:58.612816    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/kube-rbac-proxy-ovn-metrics/0.log"
May 20 16:17:58.629520 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:58.629485    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/northd/0.log"
May 20 16:17:58.647953 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:58.647907    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/nbdb/0.log"
May 20 16:17:58.667742 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:58.667715    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/sbdb/0.log"
May 20 16:17:58.878113 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:17:58.878075    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dbwqx_581482dd-fca5-4834-99f4-519a6df326fb/ovnkube-controller/0.log"
May 20 16:18:00.613855 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:18:00.613819    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-q4gpd_ddb4cb56-7904-4742-ac26-ecbd4f38df78/network-check-target-container/0.log"
May 20 16:18:01.601375 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:18:01.601347    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-sb4fl_d6731423-938b-453d-965d-db728cfa0efb/iptables-alerter/0.log"
May 20 16:18:02.340291 ip-10-0-130-227 kubenswrapper[2567]: I0520 16:18:02.340256    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-mpmsj_f65cd891-4c27-4070-972c-9a97f6fc62af/tuned/0.log"