May 25 11:22:20.208793 ip-10-0-133-58 systemd[1]: Starting Kubernetes Kubelet...
May 25 11:22:20.714191 ip-10-0-133-58 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:22:20.714191 ip-10-0-133-58 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 11:22:20.714191 ip-10-0-133-58 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:22:20.714191 ip-10-0-133-58 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 11:22:20.714191 ip-10-0-133-58 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:22:20.717188 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.717027    2578 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 11:22:20.722665 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722650    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:22:20.722665 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722665    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:22:20.722665 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722668    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722672    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722675    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722678    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722681    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722683    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722687    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722690    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722693    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722695    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722698    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722701    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722703    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722706    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722708    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722711    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722713    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722716    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722718    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722720    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:22:20.722763 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722723    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722725    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722728    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722730    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722733    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722735    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722737    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722740    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722742    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722744    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722748    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722752    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722755    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722758    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722761    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722763    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722766    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722770    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722773    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722776    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:22:20.723226 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722778    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722781    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722783    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722786    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722788    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722791    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722794    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722796    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722799    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722801    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722804    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722807    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722809    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722811    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722814    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722817    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722819    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722822    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722824    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722828    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:22:20.723761 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722831    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722833    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722836    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722838    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722840    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722844    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722846    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722849    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722852    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722854    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722857    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722860    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722862    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722865    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722867    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722869    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722872    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722875    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722879    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722881    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:22:20.724237 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722883    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722886    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722890    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.722894    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723310    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723315    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723318    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723321    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723324    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723327    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723330    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723332    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723335    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723339    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723343    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723346    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723348    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723351    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723354    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:22:20.724739 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723357    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723359    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723362    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723366    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723369    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723371    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723374    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723377    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723379    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723382    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723384    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723387    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723389    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723392    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723394    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723397    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723399    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723401    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723404    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723406    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:22:20.725190 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723409    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723411    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723414    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723416    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723419    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723422    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723424    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723427    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723429    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723432    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723434    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723437    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723440    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723443    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723445    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723448    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723451    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723469    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723471    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:22:20.725694 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723474    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723477    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723479    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723482    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723484    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723487    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723489    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723492    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723494    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723497    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723499    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723501    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723504    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723506    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723509    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723511    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723513    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723517    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723519    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723521    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:22:20.726154 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723524    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723526    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723530    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723534    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723538    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723541    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723544    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723546    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723549    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723552    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723555    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.723557    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724491    2578 flags.go:64] FLAG: --address="0.0.0.0"
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724501    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724508    2578 flags.go:64] FLAG: --anonymous-auth="true"
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724513    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724517    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724520    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724524    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724529    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724532    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 11:22:20.726650 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724535    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724539    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724542    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724545    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724548    2578 flags.go:64] FLAG: --cgroup-root=""
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724551    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724554    2578 flags.go:64] FLAG: --client-ca-file=""
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724557    2578 flags.go:64] FLAG: --cloud-config=""
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724560    2578 flags.go:64] FLAG: --cloud-provider="external"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724563    2578 flags.go:64] FLAG: --cluster-dns="[]"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724567    2578 flags.go:64] FLAG: --cluster-domain=""
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724570    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724573    2578 flags.go:64] FLAG: --config-dir=""
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724576    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724580    2578 flags.go:64] FLAG: --container-log-max-files="5"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724584    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724587    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724590    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724593    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724596    2578 flags.go:64] FLAG: --contention-profiling="false"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724599    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724602    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724606    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724609    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724613    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 11:22:20.727171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724616    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724619    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724621    2578 flags.go:64] FLAG: --enable-load-reader="false"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724625    2578 flags.go:64] FLAG: --enable-server="true"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724627    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724632    2578 flags.go:64] FLAG: --event-burst="100"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724634    2578 flags.go:64] FLAG: --event-qps="50"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724637    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724640    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724644    2578 flags.go:64] FLAG: --eviction-hard=""
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724647    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724651    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724654    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724657    2578 flags.go:64] FLAG: --eviction-soft=""
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724659    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724662    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724665    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724668    2578 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724671    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724674    2578 flags.go:64] FLAG: --fail-swap-on="true"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724677    2578 flags.go:64] FLAG: --feature-gates=""
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724681    2578 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724685    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724688    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724691    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724693    2578 flags.go:64] FLAG: --healthz-port="10248"
May 25 11:22:20.727798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724697    2578 flags.go:64] FLAG: --help="false"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724700    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-58.ec2.internal"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724703    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724706    2578 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724709    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724713    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724717    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724719    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724722    2578 flags.go:64] FLAG: --image-service-endpoint=""
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724725    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724728    2578 flags.go:64] FLAG: --kube-api-burst="100"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724731    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724734    2578 flags.go:64] FLAG: --kube-api-qps="50"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724736    2578 flags.go:64] FLAG: --kube-reserved=""
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724739    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724742    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724745    2578 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724748    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724751    2578 flags.go:64] FLAG: --lock-file=""
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724754    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724757    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724760    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724765    2578 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 11:22:20.728469 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724768    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724771    2578 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724773    2578 flags.go:64] FLAG: --logging-format="text"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724776    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724779    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724782    2578 flags.go:64] FLAG: --manifest-url=""
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724785    2578 flags.go:64] FLAG: --manifest-url-header=""
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724789    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724792    2578 flags.go:64] FLAG: --max-open-files="1000000"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724796    2578 flags.go:64] FLAG: --max-pods="110"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724799    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724802    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724805    2578 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724808    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724812    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724815    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724818    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724830    2578 flags.go:64] FLAG: --node-status-max-images="50"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724833    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724836    2578 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724839    2578 flags.go:64] FLAG: --pod-cidr=""
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724841    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724847    2578 flags.go:64] FLAG: --pod-manifest-path=""
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724850    2578 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 11:22:20.729062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724853    2578 flags.go:64] FLAG: --pods-per-core="0"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724856    2578 flags.go:64] FLAG: --port="10250"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724859    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724862    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-027dbc9d97e8bebb0"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724865    2578 flags.go:64] FLAG: --qos-reserved=""
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724868    2578 flags.go:64] FLAG: --read-only-port="10255"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724870    2578 flags.go:64] FLAG: --register-node="true"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724873    2578 flags.go:64] FLAG: --register-schedulable="true"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724876    2578 flags.go:64] FLAG: --register-with-taints=""
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724880    2578 flags.go:64] FLAG: --registry-burst="10"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724882    2578 flags.go:64] FLAG: --registry-qps="5"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724885    2578 flags.go:64] FLAG: --reserved-cpus=""
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724888    2578 flags.go:64] FLAG: --reserved-memory=""
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724892    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724895    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724898    2578 flags.go:64] FLAG: --rotate-certificates="false"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724901    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724903    2578 flags.go:64] FLAG: --runonce="false"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724906    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724909    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724912    2578 flags.go:64] FLAG: --seccomp-default="false"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724915    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724918    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724921    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724924    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724927    2578 flags.go:64] FLAG: --storage-driver-password="root"
May 25 11:22:20.729706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724930    2578 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724933    2578 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724935    2578 flags.go:64] FLAG: --storage-driver-user="root"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724938    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724941    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724944    2578 flags.go:64] FLAG: --system-cgroups=""
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724947    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724952    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724955    2578 flags.go:64] FLAG: --tls-cert-file=""
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724958    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724962    2578 flags.go:64] FLAG: --tls-min-version=""
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724965    2578 flags.go:64] FLAG: --tls-private-key-file=""
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724968    2578 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724971    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724974    2578 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724977    2578 flags.go:64] FLAG: --v="2"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724981    2578 flags.go:64] FLAG: --version="false"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724985    2578 flags.go:64] FLAG: --vmodule=""
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724989    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.724992    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725094    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725097    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725100    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725103    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:22:20.730335 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725105    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725108    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725114    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725116    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725119    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725121    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725125    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725127    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725130    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725133    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725135    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725137    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725140    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725142    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725145    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725148    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725150    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725153    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725155    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:22:20.730920 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725161    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725163    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725166    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725168    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725171    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725174    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725176    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725179    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725181    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725184    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725186    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725189    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725191    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725194    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725196    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725199    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725201    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725206    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725209    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725212    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:22:20.731408 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725216    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725218    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725221    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725223    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725226    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725229    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725231    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725234    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725237    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725240    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725242    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725245    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725248    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725251    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725254    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725257    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725259    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725261    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725264    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725266    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:22:20.731918 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725269    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725271    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725274    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725278    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725281    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725283    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725286    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725288    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725290    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725293    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725296    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725298    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725300    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725303    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725306    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725308    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725311    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725313    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725317    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:22:20.732418 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725320    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:22:20.733128 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725322    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:22:20.733128 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725325    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:22:20.733128 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.725327    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:22:20.733128 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.726196    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:22:20.735758 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.735723    2578 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 11:22:20.735758 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.735755    2578 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735810    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735815    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735819    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735823    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735825    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735828    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735831    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735834    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735836    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735839    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735842    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735844    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735847    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735850    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735852    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735855    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735857    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735859    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735862    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:22:20.735924 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735865    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735867    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735870    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735872    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735876    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735878    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735881    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735884    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735886    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735889    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735891    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735894    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735896    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735898    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735901    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735904    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735906    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735910    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735916    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:22:20.736415 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735919    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735922    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735925    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735927    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735930    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735933    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735935    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735938    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735941    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735944    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735946    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735949    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735952    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735954    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735957    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735959    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735961    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735964    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735967    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735970    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:22:20.736902 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735973    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735976    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735979    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735983    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735986    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735988    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735991    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735994    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735996    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.735999    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736002    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736004    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736007    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736009    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736011    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736014    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736016    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736019    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736022    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736025    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:22:20.737394 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736027    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736029    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736032    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736034    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736036    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736040    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736042    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736044    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.736049    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736154    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736160    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736164    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736166    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736169    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736172    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:22:20.737906 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736175    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736177    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736180    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736182    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736185    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736188    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736190    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736192    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736195    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736198    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736200    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736204    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736207    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736210    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736213    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736215    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736218    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736221    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736223    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736226    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:22:20.738278 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736228    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736230    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736233    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736235    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736238    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736240    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736243    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736245    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736248    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736250    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736253    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736255    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736258    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736260    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736263    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736265    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736268    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736270    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736273    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736275    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:22:20.738787 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736278    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736280    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736283    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736285    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736287    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736290    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736292    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736295    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736297    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736300    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736302    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736305    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736307    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736310    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736312    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736314    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736317    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736319    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736322    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736324    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:22:20.739270 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736327    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736329    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736332    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736334    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736336    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736339    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736341    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736344    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736346    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736348    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736351    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736353    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736356    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736358    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736361    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736363    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736366    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736368    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736370    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:22:20.739802 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:20.736373    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:22:20.740272 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.736377    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:22:20.740272 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.737303    2578 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 11:22:20.740864 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.740851    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 11:22:20.741907 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.741890    2578 server.go:1019] "Starting client certificate rotation"
May 25 11:22:20.742010 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.741989    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:22:20.742044 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.742031    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:22:20.769632 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.769611    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:22:20.770810 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.770790    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:22:20.792132 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.792112    2578 log.go:25] "Validated CRI v1 runtime API"
May 25 11:22:20.799976 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.799952    2578 log.go:25] "Validated CRI v1 image API"
May 25 11:22:20.802313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.802287    2578 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 11:22:20.804603 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.804582    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:22:20.804908 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.804889    2578 fs.go:135] Filesystem UUIDs: map[6958809a-fe57-48b2-b719-e86a8a64d176:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b4ebb411-090e-4878-8777-843cacbc27a4:/dev/nvme0n1p4]
May 25 11:22:20.804959 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.804910    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 11:22:20.809250 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.809136    2578 manager.go:217] Machine: {Timestamp:2026-05-25 11:22:20.808649897 +0000 UTC m=+0.457934894 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110314 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec297edd8be0bb639fc7235977f67878 SystemUUID:ec297edd-8be0-bb63-9fc7-235977f67878 BootID:9fa832bb-a424-4767-a05e-66ee4d4d60bb Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f6:d3:91:c1:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f6:d3:91:c1:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:10:4d:c4:f6:8c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 11:22:20.809250 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.809239    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 11:22:20.809368 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.809356    2578 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 11:22:20.812390 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.812366    2578 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 11:22:20.812559 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.812394    2578 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-58.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 11:22:20.812609 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.812570    2578 topology_manager.go:138] "Creating topology manager with none policy"
May 25 11:22:20.812609 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.812580    2578 container_manager_linux.go:306] "Creating device plugin manager"
May 25 11:22:20.812609 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.812587    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:22:20.813502 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.813492    2578 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:22:20.816345 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.816334    2578 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:22:20.816522 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.816512    2578 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 11:22:20.819165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.819156    2578 kubelet.go:491] "Attempting to sync node with API server"
May 25 11:22:20.819205 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.819170    2578 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 11:22:20.819205 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.819182    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 11:22:20.819205 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.819191    2578 kubelet.go:397] "Adding apiserver pod source"
May 25 11:22:20.819205 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.819199    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 11:22:20.820491 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.820478    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:22:20.820550 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.820499    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:22:20.824769 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.824749    2578 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 11:22:20.827622 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.827606    2578 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 11:22:20.828628 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.828614    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 11:22:20.828683 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.828639    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 11:22:20.828683 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.828647    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 11:22:20.828683 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.828655    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 11:22:20.828683 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.828663    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 11:22:20.828683 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.828669    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 11:22:20.828683 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.828676    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 11:22:20.828683 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.828682    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 11:22:20.828865 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.828690    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 11:22:20.828865 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.828696    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 11:22:20.828865 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.828709    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 11:22:20.829036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.829028    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 11:22:20.830147 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.830133    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 11:22:20.830147 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.830144    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 11:22:20.833386 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.833372    2578 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 11:22:20.833474 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.833407    2578 server.go:1295] "Started kubelet"
May 25 11:22:20.833720 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.833698    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 11:22:20.833806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.833772    2578 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 11:22:20.833841 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.833824    2578 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 11:22:20.834094 ip-10-0-133-58 systemd[1]: Started Kubernetes Kubelet.
May 25 11:22:20.835116 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:20.835092    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 11:22:20.835302 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:20.835280    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 11:22:20.835494 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.835452    2578 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 11:22:20.835494 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.835484    2578 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-58.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 11:22:20.836610 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.836595    2578 server.go:317] "Adding debug handlers to kubelet server"
May 25 11:22:20.845112 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:20.843948    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-58.ec2.internal.18b2cb1ff23ea7c4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-58.ec2.internal,UID:ip-10-0-133-58.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-58.ec2.internal,},FirstTimestamp:2026-05-25 11:22:20.833384388 +0000 UTC m=+0.482669375,LastTimestamp:2026-05-25 11:22:20.833384388 +0000 UTC m=+0.482669375,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-58.ec2.internal,}"
May 25 11:22:20.845439 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:20.845409    2578 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 11:22:20.848988 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.848971    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 11:22:20.849673 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.849652    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 11:22:20.849769 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.849745    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjhbx"
May 25 11:22:20.849870 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.849850    2578 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 11:22:20.849986 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.849976    2578 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 11:22:20.850115 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.849869    2578 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 11:22:20.850248 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.850214    2578 reconstruct.go:97] "Volume reconstruction finished"
May 25 11:22:20.850248 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.850231    2578 reconciler.go:26] "Reconciler: start to sync state"
May 25 11:22:20.850357 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.850282    2578 factory.go:55] Registering systemd factory
May 25 11:22:20.850357 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.850300    2578 factory.go:223] Registration of the systemd container factory successfully
May 25 11:22:20.850686 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.850567    2578 factory.go:153] Registering CRI-O factory
May 25 11:22:20.850778 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.850705    2578 factory.go:223] Registration of the crio container factory successfully
May 25 11:22:20.850778 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.850758    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 11:22:20.850778 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.850776    2578 factory.go:103] Registering Raw factory
May 25 11:22:20.850931 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:20.850588    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:20.850931 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.850791    2578 manager.go:1196] Started watching for new ooms in manager
May 25 11:22:20.851177 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.851163    2578 manager.go:319] Starting recovery of all containers
May 25 11:22:20.857883 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.857857    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjhbx"
May 25 11:22:20.859317 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:20.859249    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 25 11:22:20.859513 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:20.859359    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 11:22:20.860711 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.860690    2578 manager.go:324] Recovery completed
May 25 11:22:20.865176 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.865163    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:22:20.867310 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.867293    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:22:20.867388 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.867323    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:22:20.867388 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.867333    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientPID"
May 25 11:22:20.867856 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.867839    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 11:22:20.867856 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.867855    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 11:22:20.868009 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.867873    2578 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:22:20.871834 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.871820    2578 policy_none.go:49] "None policy: Start"
May 25 11:22:20.871905 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.871840    2578 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 11:22:20.871905 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.871853    2578 state_mem.go:35] "Initializing new in-memory state store"
May 25 11:22:20.906112 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.906092    2578 manager.go:341] "Starting Device Plugin manager"
May 25 11:22:20.906224 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:20.906128    2578 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 11:22:20.906224 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.906139    2578 server.go:85] "Starting device plugin registration server"
May 25 11:22:20.906403 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.906391    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 11:22:20.906476 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.906404    2578 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 11:22:20.906631 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.906614    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 11:22:20.906727 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.906698    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 11:22:20.906727 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.906706    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 11:22:20.907217 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:20.907193    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 11:22:20.907303 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:20.907240    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:20.928609 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.928572    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 11:22:20.941174 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.929886    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 11:22:20.941174 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.929908    2578 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 11:22:20.941174 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.929925    2578 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 11:22:20.941174 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.929932    2578 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 11:22:20.941174 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:20.930011    2578 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 11:22:20.941174 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:20.932684    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:22:21.007180 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.007143    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:22:21.008334 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.008314    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:22:21.008438 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.008350    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:22:21.008438 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.008366    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientPID"
May 25 11:22:21.008438 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.008395    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-58.ec2.internal"
May 25 11:22:21.019435 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.019416    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-58.ec2.internal"
May 25 11:22:21.019590 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.019445    2578 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-58.ec2.internal\": node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:21.031495 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.031451    2578 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-58.ec2.internal"]
May 25 11:22:21.031595 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.031536    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:22:21.032387 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.032372    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:22:21.032428 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.032404    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:22:21.032428 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.032413    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientPID"
May 25 11:22:21.034871 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.034857    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:22:21.035033 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.035018    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.035068 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.035048    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:22:21.035660 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.035642    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:22:21.035660 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.035654    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:22:21.035807 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.035677    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:22:21.035807 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.035692    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientPID"
May 25 11:22:21.035807 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.035679    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:22:21.035914 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.035843    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientPID"
May 25 11:22:21.037900 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.037884    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.037977 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.037912    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:22:21.038593 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.038573    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:22:21.038685 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.038599    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:22:21.038685 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.038608    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeHasSufficientPID"
May 25 11:22:21.044411 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.044393    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:21.051313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.051294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623ff340bd671bdb82ed443f6ccf3272-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal\" (UID: \"623ff340bd671bdb82ed443f6ccf3272\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.051373 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.051320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623ff340bd671bdb82ed443f6ccf3272-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal\" (UID: \"623ff340bd671bdb82ed443f6ccf3272\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.051373 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.051336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea7fcd2d26b56e02b20df01f9c251236-config\") pod \"kube-apiserver-proxy-ip-10-0-133-58.ec2.internal\" (UID: \"ea7fcd2d26b56e02b20df01f9c251236\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.064346 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.064322    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-58.ec2.internal\" not found" node="ip-10-0-133-58.ec2.internal"
May 25 11:22:21.069144 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.069129    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-58.ec2.internal\" not found" node="ip-10-0-133-58.ec2.internal"
May 25 11:22:21.145172 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.145133    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:21.151495 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.151439    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623ff340bd671bdb82ed443f6ccf3272-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal\" (UID: \"623ff340bd671bdb82ed443f6ccf3272\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.151495 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.151502    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623ff340bd671bdb82ed443f6ccf3272-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal\" (UID: \"623ff340bd671bdb82ed443f6ccf3272\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.151640 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.151519    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea7fcd2d26b56e02b20df01f9c251236-config\") pod \"kube-apiserver-proxy-ip-10-0-133-58.ec2.internal\" (UID: \"ea7fcd2d26b56e02b20df01f9c251236\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.151640 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.151543    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/623ff340bd671bdb82ed443f6ccf3272-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal\" (UID: \"623ff340bd671bdb82ed443f6ccf3272\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.151640 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.151583    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/623ff340bd671bdb82ed443f6ccf3272-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal\" (UID: \"623ff340bd671bdb82ed443f6ccf3272\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.151640 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.151557    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea7fcd2d26b56e02b20df01f9c251236-config\") pod \"kube-apiserver-proxy-ip-10-0-133-58.ec2.internal\" (UID: \"ea7fcd2d26b56e02b20df01f9c251236\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.245619 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.245533    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:21.346273 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.346247    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:21.368694 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.368650    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.373586 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.373451    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-58.ec2.internal"
May 25 11:22:21.447159 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.447122    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:21.547691 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.547625    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:21.648157 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.648124    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:21.742624 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.742589    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 11:22:21.743275 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.742751    2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:22:21.748746 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.748724    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:21.849394 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.849310    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:21.849394 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.849321    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 11:22:21.860443 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.860410    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 11:17:20 +0000 UTC" deadline="2027-12-17 19:25:24.587585996 +0000 UTC"
May 25 11:22:21.860443 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.860439    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13712h3m2.727150028s"
May 25 11:22:21.865065 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.865047    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:22:21.894885 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.894860    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pm9qq"
May 25 11:22:21.907167 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.907136    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pm9qq"
May 25 11:22:21.928403 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:21.928367    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623ff340bd671bdb82ed443f6ccf3272.slice/crio-f33bb86bbe5b566c1dd46beff32972f99d2f935123981f12a4108f4fb37a260a WatchSource:0}: Error finding container f33bb86bbe5b566c1dd46beff32972f99d2f935123981f12a4108f4fb37a260a: Status 404 returned error can't find the container with id f33bb86bbe5b566c1dd46beff32972f99d2f935123981f12a4108f4fb37a260a
May 25 11:22:21.928604 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:21.928585    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7fcd2d26b56e02b20df01f9c251236.slice/crio-e01841f993b8eda270e4ab9e1a8a2b1bd1868bce194448980292f161f46bf6cd WatchSource:0}: Error finding container e01841f993b8eda270e4ab9e1a8a2b1bd1868bce194448980292f161f46bf6cd: Status 404 returned error can't find the container with id e01841f993b8eda270e4ab9e1a8a2b1bd1868bce194448980292f161f46bf6cd
May 25 11:22:21.934098 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.934057    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal" event={"ID":"623ff340bd671bdb82ed443f6ccf3272","Type":"ContainerStarted","Data":"f33bb86bbe5b566c1dd46beff32972f99d2f935123981f12a4108f4fb37a260a"}
May 25 11:22:21.934223 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.934208    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:22:21.935149 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:21.935130    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-58.ec2.internal" event={"ID":"ea7fcd2d26b56e02b20df01f9c251236","Type":"ContainerStarted","Data":"e01841f993b8eda270e4ab9e1a8a2b1bd1868bce194448980292f161f46bf6cd"}
May 25 11:22:21.949851 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:21.949827    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:22.050427 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:22.050385    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:22.150876 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:22.150778    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:22.251297 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:22.251263    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-58.ec2.internal\" not found"
May 25 11:22:22.254292 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.254275    2578 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:22:22.263397 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.263372    2578 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:22:22.303573 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.303543    2578 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:22:22.350215 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.350183    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-58.ec2.internal"
May 25 11:22:22.371519 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.371488    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:22:22.372763 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.372737    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal"
May 25 11:22:22.406775 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.406699    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:22:22.556624 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.556596    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:22:22.820995 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.820921    2578 apiserver.go:52] "Watching apiserver"
May 25 11:22:22.829938 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.829912    2578 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 11:22:22.831206 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.831180    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-tpl95","openshift-network-diagnostics/network-check-target-9xqkb","openshift-image-registry/node-ca-57jwf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal","openshift-multus/multus-wdc9d","openshift-network-operator/iptables-alerter-864ms","openshift-ovn-kubernetes/ovnkube-node-pvk9r","kube-system/konnectivity-agent-6xxd2","kube-system/kube-apiserver-proxy-ip-10-0-133-58.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42","openshift-cluster-node-tuning-operator/tuned-29jf8","openshift-dns/node-resolver-wkkdr","openshift-multus/multus-additional-cni-plugins-l9jz7"]
May 25 11:22:22.833926 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.833907    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.837880 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.837862    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:22.837976 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:22.837921    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:22.837976 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.837944    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-57jwf"
May 25 11:22:22.839310 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.839288    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 11:22:22.840013 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.839992    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.841086 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.841039    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p4pbr\""
May 25 11:22:22.841238 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.841217    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 11:22:22.841442 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.841426    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 11:22:22.841684 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.841667    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p9qdl\""
May 25 11:22:22.841766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.841689    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 11:22:22.842089 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.842074    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 11:22:22.842611 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.842593    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 11:22:22.842773 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.842743    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 11:22:22.842877 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.842861    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-szzdw\""
May 25 11:22:22.842974 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.842957    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 11:22:22.843051 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.843038    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 11:22:22.843297 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.843280    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 11:22:22.844501 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.844105    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-864ms"
May 25 11:22:22.846478 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.846444    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.848714 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.848688    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:22.851095 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.851077    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:22.851181 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:22.851143    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:22.853002 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.852983    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 11:22:22.853088 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.853018    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 11:22:22.853251 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.853238    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.853648 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.853629    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 11:22:22.853755 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.853741    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 11:22:22.854307 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.854291    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 11:22:22.854418 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.854381    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 11:22:22.854677 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.854659    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dkbgs\""
May 25 11:22:22.854677 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.854673    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 11:22:22.854838 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.854817    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 11:22:22.855767 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.855560    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wkkdr"
May 25 11:22:22.855986 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.855968    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 11:22:22.856332 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.856312    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8d5hd\""
May 25 11:22:22.856570 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.856551    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cj8f4\""
May 25 11:22:22.856637 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.856587    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 11:22:22.856820 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.856803    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 11:22:22.856865 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.856819    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 11:22:22.857979 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.857962    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-khzbd\""
May 25 11:22:22.858168 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.858148    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:22.859683 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.859771 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7080cc52-e076-45dc-8653-d4e259dc912e-konnectivity-ca\") pod \"konnectivity-agent-6xxd2\" (UID: \"7080cc52-e076-45dc-8653-d4e259dc912e\") " pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:22.859771 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-socket-dir-parent\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.859771 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-var-lib-cni-bin\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.859771 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95d5a6bc-e251-4a2e-a478-debb49fb507c-iptables-alerter-script\") pod \"iptables-alerter-864ms\" (UID: \"95d5a6bc-e251-4a2e-a478-debb49fb507c\") " pod="openshift-network-operator/iptables-alerter-864ms"
May 25 11:22:22.859971 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859796    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 11:22:22.859971 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.859971 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859812    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 11:22:22.859971 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-registration-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.859971 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-daemon-config\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.859971 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-var-lib-openvswitch\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.859971 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-run-ovn\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.860267 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.859999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1575503e-b475-4021-822e-f2e700cfbf03-env-overrides\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.860267 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssnt6\" (UniqueName: \"kubernetes.io/projected/1575503e-b475-4021-822e-f2e700cfbf03-kube-api-access-ssnt6\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.860267 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7080cc52-e076-45dc-8653-d4e259dc912e-agent-certs\") pod \"konnectivity-agent-6xxd2\" (UID: \"7080cc52-e076-45dc-8653-d4e259dc912e\") " pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:22.860267 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-slash\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.860267 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-cni-dir\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.860267 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-run-openvswitch\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.860267 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1575503e-b475-4021-822e-f2e700cfbf03-ovnkube-script-lib\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.860267 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7l7c\" (UniqueName: \"kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c\") pod \"network-check-target-9xqkb\" (UID: \"d0e477d9-b58a-4766-b341-ef28fa5d69ab\") " pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:22.860267 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860270    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-os-release\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.860636 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-hostroot\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.860636 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-run-multus-certs\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.860636 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-run-ovn-kubernetes\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.860636 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-etc-selinux\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.860636 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac7c52f6-4dec-48a3-a106-0fb515461899-host\") pod \"node-ca-57jwf\" (UID: \"ac7c52f6-4dec-48a3-a106-0fb515461899\") " pod="openshift-image-registry/node-ca-57jwf"
May 25 11:22:22.860636 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e93f1f05-2be2-471a-9f26-73d201d99aca-cni-binary-copy\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.860636 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-run-netns\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.860636 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-etc-openvswitch\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.860636 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-cni-netd\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-socket-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds65w\" (UniqueName: \"kubernetes.io/projected/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-kube-api-access-ds65w\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-etc-kubernetes\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95d5a6bc-e251-4a2e-a478-debb49fb507c-host-slash\") pod \"iptables-alerter-864ms\" (UID: \"95d5a6bc-e251-4a2e-a478-debb49fb507c\") " pod="openshift-network-operator/iptables-alerter-864ms"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1575503e-b475-4021-822e-f2e700cfbf03-ovnkube-config\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-device-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-sys-fs\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-var-lib-kubelet\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-systemd-units\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-log-socket\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-conf-dir\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.860988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-node-log\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-run-systemd\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.861021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1575503e-b475-4021-822e-f2e700cfbf03-ovn-node-metrics-cert\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac7c52f6-4dec-48a3-a106-0fb515461899-serviceca\") pod \"node-ca-57jwf\" (UID: \"ac7c52f6-4dec-48a3-a106-0fb515461899\") " pod="openshift-image-registry/node-ca-57jwf"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-system-cni-dir\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-cnibin\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-var-lib-cni-multus\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fr59\" (UniqueName: \"kubernetes.io/projected/e93f1f05-2be2-471a-9f26-73d201d99aca-kube-api-access-9fr59\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-run-k8s-cni-cncf-io\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvhws\" (UniqueName: \"kubernetes.io/projected/95d5a6bc-e251-4a2e-a478-debb49fb507c-kube-api-access-nvhws\") pod \"iptables-alerter-864ms\" (UID: \"95d5a6bc-e251-4a2e-a478-debb49fb507c\") " pod="openshift-network-operator/iptables-alerter-864ms"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-run-netns\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc8mp\" (UniqueName: \"kubernetes.io/projected/ac7c52f6-4dec-48a3-a106-0fb515461899-kube-api-access-vc8mp\") pod \"node-ca-57jwf\" (UID: \"ac7c52f6-4dec-48a3-a106-0fb515461899\") " pod="openshift-image-registry/node-ca-57jwf"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh8pz\" (UniqueName: \"kubernetes.io/projected/489b5117-49e0-45e5-b99a-3ad57febc838-kube-api-access-qh8pz\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-kubelet\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861414    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6bphw\""
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-cni-bin\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.861701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.861691    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 11:22:22.862359 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.862341    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4nhb9\""
May 25 11:22:22.862509 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.862490    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 11:22:22.862661 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.862643    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 11:22:22.908991 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.908964    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:17:21 +0000 UTC" deadline="2028-01-23 03:09:24.882200902 +0000 UTC"
May 25 11:22:22.908991 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.908988    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14583h47m1.97321467s"
May 25 11:22:22.951437 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.951404    2578 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 11:22:22.961861 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.961825    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-cni-dir\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.962002 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.961874    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-run-openvswitch\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.962002 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.961893    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1575503e-b475-4021-822e-f2e700cfbf03-ovnkube-script-lib\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.962002 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.961912    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-sysctl-d\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.962002 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.961934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-run\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.962002 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.961935    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-cni-dir\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.962002 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.961950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-cnibin\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:22.962002 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.961987    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l7c\" (UniqueName: \"kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c\") pod \"network-check-target-9xqkb\" (UID: \"d0e477d9-b58a-4766-b341-ef28fa5d69ab\") " pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962008    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-os-release\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962053    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-run-openvswitch\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962079    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-hostroot\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962107    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-os-release\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962120    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-run-multus-certs\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962146    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-hostroot\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962156    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-run-ovn-kubernetes\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962189    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-modprobe-d\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962192    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-run-ovn-kubernetes\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962150    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-run-multus-certs\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962215    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-etc-selinux\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962241    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac7c52f6-4dec-48a3-a106-0fb515461899-host\") pod \"node-ca-57jwf\" (UID: \"ac7c52f6-4dec-48a3-a106-0fb515461899\") " pod="openshift-image-registry/node-ca-57jwf"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962252    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-etc-selinux\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962265    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e93f1f05-2be2-471a-9f26-73d201d99aca-cni-binary-copy\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962302    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac7c52f6-4dec-48a3-a106-0fb515461899-host\") pod \"node-ca-57jwf\" (UID: \"ac7c52f6-4dec-48a3-a106-0fb515461899\") " pod="openshift-image-registry/node-ca-57jwf"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962289    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-run-netns\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962343    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-etc-openvswitch\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.962358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962366    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-cni-netd\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-os-release\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962402    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-run-netns\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962412    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-etc-openvswitch\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62c640d4-74b5-4e3c-b772-f4e818c25bfd-cni-binary-copy\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962480    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-socket-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962438    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-cni-netd\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962507    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ds65w\" (UniqueName: \"kubernetes.io/projected/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-kube-api-access-ds65w\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962532    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-etc-kubernetes\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962569    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95d5a6bc-e251-4a2e-a478-debb49fb507c-host-slash\") pod \"iptables-alerter-864ms\" (UID: \"95d5a6bc-e251-4a2e-a478-debb49fb507c\") " pod="openshift-network-operator/iptables-alerter-864ms"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962593    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1575503e-b475-4021-822e-f2e700cfbf03-ovnkube-config\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-var-lib-kubelet\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjj6\" (UniqueName: \"kubernetes.io/projected/c4ac3548-3f56-4896-8325-5a69d47b6008-kube-api-access-xdjj6\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962649    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1575503e-b475-4021-822e-f2e700cfbf03-ovnkube-script-lib\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962658    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95d5a6bc-e251-4a2e-a478-debb49fb507c-host-slash\") pod \"iptables-alerter-864ms\" (UID: \"95d5a6bc-e251-4a2e-a478-debb49fb507c\") " pod="openshift-network-operator/iptables-alerter-864ms"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962684    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-etc-kubernetes\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962693    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-socket-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.963122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962756    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-device-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962782    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-sys-fs\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962798    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-var-lib-kubelet\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962807    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-device-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962816    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-systemd-units\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962840    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-log-socket\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962852    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-var-lib-kubelet\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962863    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-sys-fs\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962869    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkg9q\" (UniqueName: \"kubernetes.io/projected/62c640d4-74b5-4e3c-b772-f4e818c25bfd-kube-api-access-xkg9q\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962898    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-systemd-units\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962902    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-log-socket\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962933    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-conf-dir\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962962    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-node-log\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4ac3548-3f56-4896-8325-5a69d47b6008-tmp\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962984    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-conf-dir\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.962996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-system-cni-dir\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963011    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-node-log\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.963902 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:22.962988    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6375e270-ce2d-4549-a415-343be34c4dfc-hosts-file\") pod \"node-resolver-wkkdr\" (UID: \"6375e270-ce2d-4549-a415-343be34c4dfc\") " pod="openshift-dns/node-resolver-wkkdr"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963053    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-run-systemd\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-kubernetes\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963112    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-run-systemd\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1575503e-b475-4021-822e-f2e700cfbf03-ovnkube-config\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-tuned\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:22.963165    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs podName:489b5117-49e0-45e5-b99a-3ad57febc838 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:23.463133428 +0000 UTC m=+3.112418401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs") pod "network-metrics-daemon-tpl95" (UID: "489b5117-49e0-45e5-b99a-3ad57febc838") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963233    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1575503e-b475-4021-822e-f2e700cfbf03-ovn-node-metrics-cert\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-sysconfig\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963275    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac7c52f6-4dec-48a3-a106-0fb515461899-serviceca\") pod \"node-ca-57jwf\" (UID: \"ac7c52f6-4dec-48a3-a106-0fb515461899\") " pod="openshift-image-registry/node-ca-57jwf"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963292    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-system-cni-dir\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963307    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-cnibin\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62c640d4-74b5-4e3c-b772-f4e818c25bfd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963365    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-cnibin\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963381    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-var-lib-cni-multus\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.964638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963401    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9fr59\" (UniqueName: \"kubernetes.io/projected/e93f1f05-2be2-471a-9f26-73d201d99aca-kube-api-access-9fr59\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4mt\" (UniqueName: \"kubernetes.io/projected/6375e270-ce2d-4549-a415-343be34c4dfc-kube-api-access-gw4mt\") pod \"node-resolver-wkkdr\" (UID: \"6375e270-ce2d-4549-a415-343be34c4dfc\") " pod="openshift-dns/node-resolver-wkkdr"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963445    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-run-k8s-cni-cncf-io\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963418    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-system-cni-dir\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963487    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvhws\" (UniqueName: \"kubernetes.io/projected/95d5a6bc-e251-4a2e-a478-debb49fb507c-kube-api-access-nvhws\") pod \"iptables-alerter-864ms\" (UID: \"95d5a6bc-e251-4a2e-a478-debb49fb507c\") " pod="openshift-network-operator/iptables-alerter-864ms"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963348    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e93f1f05-2be2-471a-9f26-73d201d99aca-cni-binary-copy\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963480    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-var-lib-cni-multus\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963516    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-run-k8s-cni-cncf-io\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963528    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-run-netns\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963561    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vc8mp\" (UniqueName: \"kubernetes.io/projected/ac7c52f6-4dec-48a3-a106-0fb515461899-kube-api-access-vc8mp\") pod \"node-ca-57jwf\" (UID: \"ac7c52f6-4dec-48a3-a106-0fb515461899\") " pod="openshift-image-registry/node-ca-57jwf"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963592    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qh8pz\" (UniqueName: \"kubernetes.io/projected/489b5117-49e0-45e5-b99a-3ad57febc838-kube-api-access-qh8pz\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963608    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963617    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-run-netns\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-sys\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-lib-modules\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963796    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-kubelet\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963819    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac7c52f6-4dec-48a3-a106-0fb515461899-serviceca\") pod \"node-ca-57jwf\" (UID: \"ac7c52f6-4dec-48a3-a106-0fb515461899\") " pod="openshift-image-registry/node-ca-57jwf"
May 25 11:22:22.965165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963821    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-cni-bin\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963861    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-kubelet\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963874    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-cni-bin\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6375e270-ce2d-4549-a415-343be34c4dfc-tmp-dir\") pod \"node-resolver-wkkdr\" (UID: \"6375e270-ce2d-4549-a415-343be34c4dfc\") " pod="openshift-dns/node-resolver-wkkdr"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963912    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963935    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7080cc52-e076-45dc-8653-d4e259dc912e-konnectivity-ca\") pod \"konnectivity-agent-6xxd2\" (UID: \"7080cc52-e076-45dc-8653-d4e259dc912e\") " pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.963984    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964012    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-socket-dir-parent\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964040    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-var-lib-cni-bin\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964065    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95d5a6bc-e251-4a2e-a478-debb49fb507c-iptables-alerter-script\") pod \"iptables-alerter-864ms\" (UID: \"95d5a6bc-e251-4a2e-a478-debb49fb507c\") " pod="openshift-network-operator/iptables-alerter-864ms"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964076    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-socket-dir-parent\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964092    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964096    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e93f1f05-2be2-471a-9f26-73d201d99aca-host-var-lib-cni-bin\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-sysctl-conf\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62c640d4-74b5-4e3c-b772-f4e818c25bfd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964150    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964182    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-registration-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.965806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964209    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-daemon-config\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964217    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-registration-dir\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964234    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-var-lib-openvswitch\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964262    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-run-ovn\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964286    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1575503e-b475-4021-822e-f2e700cfbf03-env-overrides\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964312    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ssnt6\" (UniqueName: \"kubernetes.io/projected/1575503e-b475-4021-822e-f2e700cfbf03-kube-api-access-ssnt6\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964325    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-var-lib-openvswitch\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-systemd\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964417    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7080cc52-e076-45dc-8653-d4e259dc912e-agent-certs\") pod \"konnectivity-agent-6xxd2\" (UID: \"7080cc52-e076-45dc-8653-d4e259dc912e\") " pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964443    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7080cc52-e076-45dc-8653-d4e259dc912e-konnectivity-ca\") pod \"konnectivity-agent-6xxd2\" (UID: \"7080cc52-e076-45dc-8653-d4e259dc912e\") " pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964481    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-slash\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-host\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964529    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-host-slash\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964616    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95d5a6bc-e251-4a2e-a478-debb49fb507c-iptables-alerter-script\") pod \"iptables-alerter-864ms\" (UID: \"95d5a6bc-e251-4a2e-a478-debb49fb507c\") " pod="openshift-network-operator/iptables-alerter-864ms"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964666    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1575503e-b475-4021-822e-f2e700cfbf03-run-ovn\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964689    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e93f1f05-2be2-471a-9f26-73d201d99aca-multus-daemon-config\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.966614 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.964772    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1575503e-b475-4021-822e-f2e700cfbf03-env-overrides\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.967319 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.966802    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1575503e-b475-4021-822e-f2e700cfbf03-ovn-node-metrics-cert\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.967319 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.967021    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7080cc52-e076-45dc-8653-d4e259dc912e-agent-certs\") pod \"konnectivity-agent-6xxd2\" (UID: \"7080cc52-e076-45dc-8653-d4e259dc912e\") " pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:22.976051 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:22.976029    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:22:22.976051 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:22.976055    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:22:22.976225 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:22.976069    2578 projected.go:194] Error preparing data for projected volume kube-api-access-z7l7c for pod openshift-network-diagnostics/network-check-target-9xqkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:22.976225 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:22.976128    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c podName:d0e477d9-b58a-4766-b341-ef28fa5d69ab nodeName:}" failed. No retries permitted until 2026-05-25 11:22:23.476110971 +0000 UTC m=+3.125395945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z7l7c" (UniqueName: "kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c") pod "network-check-target-9xqkb" (UID: "d0e477d9-b58a-4766-b341-ef28fa5d69ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:22.979582 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.979560    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fr59\" (UniqueName: \"kubernetes.io/projected/e93f1f05-2be2-471a-9f26-73d201d99aca-kube-api-access-9fr59\") pod \"multus-wdc9d\" (UID: \"e93f1f05-2be2-471a-9f26-73d201d99aca\") " pod="openshift-multus/multus-wdc9d"
May 25 11:22:22.979732 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.979707    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvhws\" (UniqueName: \"kubernetes.io/projected/95d5a6bc-e251-4a2e-a478-debb49fb507c-kube-api-access-nvhws\") pod \"iptables-alerter-864ms\" (UID: \"95d5a6bc-e251-4a2e-a478-debb49fb507c\") " pod="openshift-network-operator/iptables-alerter-864ms"
May 25 11:22:22.983647 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.983623    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds65w\" (UniqueName: \"kubernetes.io/projected/7f10ac15-85c6-4bca-8d2d-d65bc5d13456-kube-api-access-ds65w\") pod \"aws-ebs-csi-driver-node-ngd42\" (UID: \"7f10ac15-85c6-4bca-8d2d-d65bc5d13456\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:22.984024 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.983997    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc8mp\" (UniqueName: \"kubernetes.io/projected/ac7c52f6-4dec-48a3-a106-0fb515461899-kube-api-access-vc8mp\") pod \"node-ca-57jwf\" (UID: \"ac7c52f6-4dec-48a3-a106-0fb515461899\") " pod="openshift-image-registry/node-ca-57jwf"
May 25 11:22:22.984564 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.984541    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssnt6\" (UniqueName: \"kubernetes.io/projected/1575503e-b475-4021-822e-f2e700cfbf03-kube-api-access-ssnt6\") pod \"ovnkube-node-pvk9r\" (UID: \"1575503e-b475-4021-822e-f2e700cfbf03\") " pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:22.984654 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:22.984606    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh8pz\" (UniqueName: \"kubernetes.io/projected/489b5117-49e0-45e5-b99a-3ad57febc838-kube-api-access-qh8pz\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:23.065737 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065710    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-sysconfig\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.065737 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065743    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62c640d4-74b5-4e3c-b772-f4e818c25bfd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.065973 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065768    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4mt\" (UniqueName: \"kubernetes.io/projected/6375e270-ce2d-4549-a415-343be34c4dfc-kube-api-access-gw4mt\") pod \"node-resolver-wkkdr\" (UID: \"6375e270-ce2d-4549-a415-343be34c4dfc\") " pod="openshift-dns/node-resolver-wkkdr"
May 25 11:22:23.065973 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065788    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-sys\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.065973 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065806    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-lib-modules\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.065973 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065833    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-sysconfig\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.065973 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065878    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.065973 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065890    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-sys\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.065973 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065906    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6375e270-ce2d-4549-a415-343be34c4dfc-tmp-dir\") pod \"node-resolver-wkkdr\" (UID: \"6375e270-ce2d-4549-a415-343be34c4dfc\") " pod="openshift-dns/node-resolver-wkkdr"
May 25 11:22:23.065973 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065919    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-lib-modules\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.065973 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065949    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-sysctl-conf\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.065979    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62c640d4-74b5-4e3c-b772-f4e818c25bfd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066011    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-systemd\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066038    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-host\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066059    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066074    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-sysctl-d\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066081    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-sysctl-conf\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066098    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-run\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066109    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-systemd\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066129    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-cnibin\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066136    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-host\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066171    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-modprobe-d\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066177    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6375e270-ce2d-4549-a415-343be34c4dfc-tmp-dir\") pod \"node-resolver-wkkdr\" (UID: \"6375e270-ce2d-4549-a415-343be34c4dfc\") " pod="openshift-dns/node-resolver-wkkdr"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066185    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-run\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066202    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-os-release\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066214    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-sysctl-d\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066222    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-cnibin\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066226    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62c640d4-74b5-4e3c-b772-f4e818c25bfd-cni-binary-copy\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066256    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-var-lib-kubelet\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066270    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-modprobe-d\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-os-release\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066281    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjj6\" (UniqueName: \"kubernetes.io/projected/c4ac3548-3f56-4896-8325-5a69d47b6008-kube-api-access-xdjj6\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066320    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-var-lib-kubelet\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066323    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xkg9q\" (UniqueName: \"kubernetes.io/projected/62c640d4-74b5-4e3c-b772-f4e818c25bfd-kube-api-access-xkg9q\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066350    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4ac3548-3f56-4896-8325-5a69d47b6008-tmp\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066374    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-system-cni-dir\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066398    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6375e270-ce2d-4549-a415-343be34c4dfc-hosts-file\") pod \"node-resolver-wkkdr\" (UID: \"6375e270-ce2d-4549-a415-343be34c4dfc\") " pod="openshift-dns/node-resolver-wkkdr"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066423    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-kubernetes\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066427    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/62c640d4-74b5-4e3c-b772-f4e818c25bfd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066446    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-tuned\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066513    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/62c640d4-74b5-4e3c-b772-f4e818c25bfd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066552    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c640d4-74b5-4e3c-b772-f4e818c25bfd-system-cni-dir\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066561    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-kubernetes\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066584    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6375e270-ce2d-4549-a415-343be34c4dfc-hosts-file\") pod \"node-resolver-wkkdr\" (UID: \"6375e270-ce2d-4549-a415-343be34c4dfc\") " pod="openshift-dns/node-resolver-wkkdr"
May 25 11:22:23.066984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.066695    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62c640d4-74b5-4e3c-b772-f4e818c25bfd-cni-binary-copy\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.068686 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.068661    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4ac3548-3f56-4896-8325-5a69d47b6008-tmp\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.068846 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.068694    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4ac3548-3f56-4896-8325-5a69d47b6008-etc-tuned\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.079952 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.079874    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjj6\" (UniqueName: \"kubernetes.io/projected/c4ac3548-3f56-4896-8325-5a69d47b6008-kube-api-access-xdjj6\") pod \"tuned-29jf8\" (UID: \"c4ac3548-3f56-4896-8325-5a69d47b6008\") " pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.080860 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.080836    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4mt\" (UniqueName: \"kubernetes.io/projected/6375e270-ce2d-4549-a415-343be34c4dfc-kube-api-access-gw4mt\") pod \"node-resolver-wkkdr\" (UID: \"6375e270-ce2d-4549-a415-343be34c4dfc\") " pod="openshift-dns/node-resolver-wkkdr"
May 25 11:22:23.082056 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.082034    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkg9q\" (UniqueName: \"kubernetes.io/projected/62c640d4-74b5-4e3c-b772-f4e818c25bfd-kube-api-access-xkg9q\") pod \"multus-additional-cni-plugins-l9jz7\" (UID: \"62c640d4-74b5-4e3c-b772-f4e818c25bfd\") " pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.145820 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.145786    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42"
May 25 11:22:23.154713 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.154681    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-57jwf"
May 25 11:22:23.166450 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.166421    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wdc9d"
May 25 11:22:23.170119 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.170095    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-864ms"
May 25 11:22:23.176752 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.176732    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:23.183281 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.183263    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:23.190816 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.190795    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-29jf8"
May 25 11:22:23.197422 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.197402    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wkkdr"
May 25 11:22:23.202018 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.202000    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l9jz7"
May 25 11:22:23.469122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.469041    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:23.469290 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:23.469189    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:23.469290 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:23.469257    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs podName:489b5117-49e0-45e5-b99a-3ad57febc838 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:24.469236003 +0000 UTC m=+4.118520980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs") pod "network-metrics-daemon-tpl95" (UID: "489b5117-49e0-45e5-b99a-3ad57febc838") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:23.569380 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.569344    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l7c\" (UniqueName: \"kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c\") pod \"network-check-target-9xqkb\" (UID: \"d0e477d9-b58a-4766-b341-ef28fa5d69ab\") " pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:23.569541 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:23.569509    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:22:23.569541 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:23.569531    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:22:23.569645 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:23.569543    2578 projected.go:194] Error preparing data for projected volume kube-api-access-z7l7c for pod openshift-network-diagnostics/network-check-target-9xqkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:23.569645 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:23.569606    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c podName:d0e477d9-b58a-4766-b341-ef28fa5d69ab nodeName:}" failed. No retries permitted until 2026-05-25 11:22:24.569587069 +0000 UTC m=+4.218872047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7l7c" (UniqueName: "kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c") pod "network-check-target-9xqkb" (UID: "d0e477d9-b58a-4766-b341-ef28fa5d69ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:23.608953 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:23.608886    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7080cc52_e076_45dc_8653_d4e259dc912e.slice/crio-f6cdd49033bd1f4b796c33b8d2e41b3d7d5d4305ca3e7c1982f60403a1e5b808 WatchSource:0}: Error finding container f6cdd49033bd1f4b796c33b8d2e41b3d7d5d4305ca3e7c1982f60403a1e5b808: Status 404 returned error can't find the container with id f6cdd49033bd1f4b796c33b8d2e41b3d7d5d4305ca3e7c1982f60403a1e5b808
May 25 11:22:23.610446 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:23.610288    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f10ac15_85c6_4bca_8d2d_d65bc5d13456.slice/crio-8a8b4be18a34f395b4669f24e7dc10b08ed3bcdf7b197b9360c0cc11f1aeb20d WatchSource:0}: Error finding container 8a8b4be18a34f395b4669f24e7dc10b08ed3bcdf7b197b9360c0cc11f1aeb20d: Status 404 returned error can't find the container with id 8a8b4be18a34f395b4669f24e7dc10b08ed3bcdf7b197b9360c0cc11f1aeb20d
May 25 11:22:23.611296 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:23.611274    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6375e270_ce2d_4549_a415_343be34c4dfc.slice/crio-7d9f7ab8ce22abea61b1805eaaf761b0d6161d306bb237d6faec305cd9fb646b WatchSource:0}: Error finding container 7d9f7ab8ce22abea61b1805eaaf761b0d6161d306bb237d6faec305cd9fb646b: Status 404 returned error can't find the container with id 7d9f7ab8ce22abea61b1805eaaf761b0d6161d306bb237d6faec305cd9fb646b
May 25 11:22:23.614862 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:23.614843    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d5a6bc_e251_4a2e_a478_debb49fb507c.slice/crio-67918f10f85dd8ddf409cb9897ec38987b85257fe31310311111e0b979a860c4 WatchSource:0}: Error finding container 67918f10f85dd8ddf409cb9897ec38987b85257fe31310311111e0b979a860c4: Status 404 returned error can't find the container with id 67918f10f85dd8ddf409cb9897ec38987b85257fe31310311111e0b979a860c4
May 25 11:22:23.615323 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:23.615303    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1575503e_b475_4021_822e_f2e700cfbf03.slice/crio-65a46127182e8805400956d8b9660c2ed4ac133897b85402eb1f8003e61d27fd WatchSource:0}: Error finding container 65a46127182e8805400956d8b9660c2ed4ac133897b85402eb1f8003e61d27fd: Status 404 returned error can't find the container with id 65a46127182e8805400956d8b9660c2ed4ac133897b85402eb1f8003e61d27fd
May 25 11:22:23.616373 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:23.616352    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93f1f05_2be2_471a_9f26_73d201d99aca.slice/crio-39c6ff0182733a86db61913a4b801cea66be39c2ec77dc1014539678339598c4 WatchSource:0}: Error finding container 39c6ff0182733a86db61913a4b801cea66be39c2ec77dc1014539678339598c4: Status 404 returned error can't find the container with id 39c6ff0182733a86db61913a4b801cea66be39c2ec77dc1014539678339598c4
May 25 11:22:23.617222 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:23.617191    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7c52f6_4dec_48a3_a106_0fb515461899.slice/crio-6c14740d9ca505ed8036d4219890571cb8a32f23bfd6ee9921ee7441dc281827 WatchSource:0}: Error finding container 6c14740d9ca505ed8036d4219890571cb8a32f23bfd6ee9921ee7441dc281827: Status 404 returned error can't find the container with id 6c14740d9ca505ed8036d4219890571cb8a32f23bfd6ee9921ee7441dc281827
May 25 11:22:23.621398 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:23.621377    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c640d4_74b5_4e3c_b772_f4e818c25bfd.slice/crio-b524dd6202e1a8ffa72062b64a59d506bebc41f8dc4a2dfd9a9162790bebe694 WatchSource:0}: Error finding container b524dd6202e1a8ffa72062b64a59d506bebc41f8dc4a2dfd9a9162790bebe694: Status 404 returned error can't find the container with id b524dd6202e1a8ffa72062b64a59d506bebc41f8dc4a2dfd9a9162790bebe694
May 25 11:22:23.622659 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:23.622614    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ac3548_3f56_4896_8325_5a69d47b6008.slice/crio-6ac5e0a7019878dbaefc4dfebb4883813a0a6bdd37ba0a12822427ff37ff4dac WatchSource:0}: Error finding container 6ac5e0a7019878dbaefc4dfebb4883813a0a6bdd37ba0a12822427ff37ff4dac: Status 404 returned error can't find the container with id 6ac5e0a7019878dbaefc4dfebb4883813a0a6bdd37ba0a12822427ff37ff4dac
May 25 11:22:23.910127 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.909972    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:17:21 +0000 UTC" deadline="2027-10-18 15:46:13.144342124 +0000 UTC"
May 25 11:22:23.910127 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.910127    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12268h23m49.234218566s"
May 25 11:22:23.939592 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.939551    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal" event={"ID":"623ff340bd671bdb82ed443f6ccf3272","Type":"ContainerStarted","Data":"e79f39d1f4983b3776cfacc6ad504c96107ff56a00364762f5e2c8efc1b6db0f"}
May 25 11:22:23.940701 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.940672    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-29jf8" event={"ID":"c4ac3548-3f56-4896-8325-5a69d47b6008","Type":"ContainerStarted","Data":"6ac5e0a7019878dbaefc4dfebb4883813a0a6bdd37ba0a12822427ff37ff4dac"}
May 25 11:22:23.941826 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.941804    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r" event={"ID":"1575503e-b475-4021-822e-f2e700cfbf03","Type":"ContainerStarted","Data":"65a46127182e8805400956d8b9660c2ed4ac133897b85402eb1f8003e61d27fd"}
May 25 11:22:23.942825 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.942736    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wkkdr" event={"ID":"6375e270-ce2d-4549-a415-343be34c4dfc","Type":"ContainerStarted","Data":"7d9f7ab8ce22abea61b1805eaaf761b0d6161d306bb237d6faec305cd9fb646b"}
May 25 11:22:23.943727 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.943703    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42" event={"ID":"7f10ac15-85c6-4bca-8d2d-d65bc5d13456","Type":"ContainerStarted","Data":"8a8b4be18a34f395b4669f24e7dc10b08ed3bcdf7b197b9360c0cc11f1aeb20d"}
May 25 11:22:23.945062 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.945040    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-58.ec2.internal" event={"ID":"ea7fcd2d26b56e02b20df01f9c251236","Type":"ContainerStarted","Data":"0195ed65924b462662df473c11d95d4d6f74c1d3a3b2d876261bf97274de5f26"}
May 25 11:22:23.946110 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.946086    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9jz7" event={"ID":"62c640d4-74b5-4e3c-b772-f4e818c25bfd","Type":"ContainerStarted","Data":"b524dd6202e1a8ffa72062b64a59d506bebc41f8dc4a2dfd9a9162790bebe694"}
May 25 11:22:23.947056 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.947027    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-57jwf" event={"ID":"ac7c52f6-4dec-48a3-a106-0fb515461899","Type":"ContainerStarted","Data":"6c14740d9ca505ed8036d4219890571cb8a32f23bfd6ee9921ee7441dc281827"}
May 25 11:22:23.948003 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.947985    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wdc9d" event={"ID":"e93f1f05-2be2-471a-9f26-73d201d99aca","Type":"ContainerStarted","Data":"39c6ff0182733a86db61913a4b801cea66be39c2ec77dc1014539678339598c4"}
May 25 11:22:23.948979 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.948954    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-864ms" event={"ID":"95d5a6bc-e251-4a2e-a478-debb49fb507c","Type":"ContainerStarted","Data":"67918f10f85dd8ddf409cb9897ec38987b85257fe31310311111e0b979a860c4"}
May 25 11:22:23.949972 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.949954    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6xxd2" event={"ID":"7080cc52-e076-45dc-8653-d4e259dc912e","Type":"ContainerStarted","Data":"f6cdd49033bd1f4b796c33b8d2e41b3d7d5d4305ca3e7c1982f60403a1e5b808"}
May 25 11:22:23.970536 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:23.970483    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-58.ec2.internal" podStartSLOduration=1.970448331 podStartE2EDuration="1.970448331s" podCreationTimestamp="2026-05-25 11:22:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:22:23.970073815 +0000 UTC m=+3.619358813" watchObservedRunningTime="2026-05-25 11:22:23.970448331 +0000 UTC m=+3.619733329"
May 25 11:22:24.476014 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:24.475972    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:24.476202 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:24.476164    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:24.476265 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:24.476227    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs podName:489b5117-49e0-45e5-b99a-3ad57febc838 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:26.476208263 +0000 UTC m=+6.125493237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs") pod "network-metrics-daemon-tpl95" (UID: "489b5117-49e0-45e5-b99a-3ad57febc838") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:24.577020 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:24.576981    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l7c\" (UniqueName: \"kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c\") pod \"network-check-target-9xqkb\" (UID: \"d0e477d9-b58a-4766-b341-ef28fa5d69ab\") " pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:24.577243 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:24.577192    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:22:24.577243 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:24.577218    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:22:24.577243 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:24.577233    2578 projected.go:194] Error preparing data for projected volume kube-api-access-z7l7c for pod openshift-network-diagnostics/network-check-target-9xqkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:24.577416 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:24.577298    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c podName:d0e477d9-b58a-4766-b341-ef28fa5d69ab nodeName:}" failed. No retries permitted until 2026-05-25 11:22:26.577278956 +0000 UTC m=+6.226563934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7l7c" (UniqueName: "kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c") pod "network-check-target-9xqkb" (UID: "d0e477d9-b58a-4766-b341-ef28fa5d69ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:24.932986 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:24.932898    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:24.933414 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:24.933017    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:24.933514 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:24.933424    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:24.933576 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:24.933541    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:24.964243 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:24.964207    2578 generic.go:358] "Generic (PLEG): container finished" podID="623ff340bd671bdb82ed443f6ccf3272" containerID="e79f39d1f4983b3776cfacc6ad504c96107ff56a00364762f5e2c8efc1b6db0f" exitCode=0
May 25 11:22:24.965166 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:24.965138    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal" event={"ID":"623ff340bd671bdb82ed443f6ccf3272","Type":"ContainerDied","Data":"e79f39d1f4983b3776cfacc6ad504c96107ff56a00364762f5e2c8efc1b6db0f"}
May 25 11:22:25.975099 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:25.975010    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal" event={"ID":"623ff340bd671bdb82ed443f6ccf3272","Type":"ContainerStarted","Data":"8a8efbffb7a47cd283080b7687bbdf190f7c199f3d6af453fd6c6b140885ffd2"}
May 25 11:22:26.491919 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:26.491881    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:26.492105 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:26.492051    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:26.492179 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:26.492115    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs podName:489b5117-49e0-45e5-b99a-3ad57febc838 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:30.492096409 +0000 UTC m=+10.141381390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs") pod "network-metrics-daemon-tpl95" (UID: "489b5117-49e0-45e5-b99a-3ad57febc838") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:26.592635 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:26.592592    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l7c\" (UniqueName: \"kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c\") pod \"network-check-target-9xqkb\" (UID: \"d0e477d9-b58a-4766-b341-ef28fa5d69ab\") " pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:26.592852 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:26.592806    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:22:26.592852 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:26.592831    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:22:26.592852 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:26.592843    2578 projected.go:194] Error preparing data for projected volume kube-api-access-z7l7c for pod openshift-network-diagnostics/network-check-target-9xqkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:26.593030 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:26.592903    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c podName:d0e477d9-b58a-4766-b341-ef28fa5d69ab nodeName:}" failed. No retries permitted until 2026-05-25 11:22:30.592885398 +0000 UTC m=+10.242170373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7l7c" (UniqueName: "kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c") pod "network-check-target-9xqkb" (UID: "d0e477d9-b58a-4766-b341-ef28fa5d69ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:26.931671 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:26.931587    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:26.931824 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:26.931598    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:26.931824 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:26.931760    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:26.931824 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:26.931803    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:28.933372 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:28.932712    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:28.933372 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:28.932833    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:28.933372 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:28.933202    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:28.933372 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:28.933316    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:29.059068 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.058973    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-58.ec2.internal" podStartSLOduration=7.058953715 podStartE2EDuration="7.058953715s" podCreationTimestamp="2026-05-25 11:22:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:22:25.990097786 +0000 UTC m=+5.639382788" watchObservedRunningTime="2026-05-25 11:22:29.058953715 +0000 UTC m=+8.708238713"
May 25 11:22:29.059873 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.059853    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-vcnp5"]
May 25 11:22:29.062924 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.062618    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:29.062924 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:29.062703    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:29.114707 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.114667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d037290-6471-4e48-bf2e-bcaa7de9cc64-dbus\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:29.114887 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.114725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:29.114887 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.114778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d037290-6471-4e48-bf2e-bcaa7de9cc64-kubelet-config\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:29.215609 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.215315    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d037290-6471-4e48-bf2e-bcaa7de9cc64-dbus\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:29.215609 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.215373    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:29.215609 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.215416    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d037290-6471-4e48-bf2e-bcaa7de9cc64-kubelet-config\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:29.215609 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.215520    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d037290-6471-4e48-bf2e-bcaa7de9cc64-kubelet-config\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:29.215609 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.215520    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d037290-6471-4e48-bf2e-bcaa7de9cc64-dbus\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:29.215609 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:29.215573    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:22:29.215949 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:29.215644    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret podName:9d037290-6471-4e48-bf2e-bcaa7de9cc64 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:29.715625236 +0000 UTC m=+9.364910209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret") pod "global-pull-secret-syncer-vcnp5" (UID: "9d037290-6471-4e48-bf2e-bcaa7de9cc64") : object "kube-system"/"original-pull-secret" not registered
May 25 11:22:29.719373 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:29.719332    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:29.719571 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:29.719486    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:22:29.719571 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:29.719548    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret podName:9d037290-6471-4e48-bf2e-bcaa7de9cc64 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:30.719530352 +0000 UTC m=+10.368815346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret") pod "global-pull-secret-syncer-vcnp5" (UID: "9d037290-6471-4e48-bf2e-bcaa7de9cc64") : object "kube-system"/"original-pull-secret" not registered
May 25 11:22:30.525975 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:30.525924    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:30.526391 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:30.526045    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:30.526391 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:30.526092    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs podName:489b5117-49e0-45e5-b99a-3ad57febc838 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:38.526077008 +0000 UTC m=+18.175361986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs") pod "network-metrics-daemon-tpl95" (UID: "489b5117-49e0-45e5-b99a-3ad57febc838") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:30.627257 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:30.627214    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l7c\" (UniqueName: \"kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c\") pod \"network-check-target-9xqkb\" (UID: \"d0e477d9-b58a-4766-b341-ef28fa5d69ab\") " pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:30.627437 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:30.627361    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:22:30.627437 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:30.627379    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:22:30.627437 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:30.627390    2578 projected.go:194] Error preparing data for projected volume kube-api-access-z7l7c for pod openshift-network-diagnostics/network-check-target-9xqkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:30.627628 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:30.627447    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c podName:d0e477d9-b58a-4766-b341-ef28fa5d69ab nodeName:}" failed. No retries permitted until 2026-05-25 11:22:38.627429338 +0000 UTC m=+18.276714311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7l7c" (UniqueName: "kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c") pod "network-check-target-9xqkb" (UID: "d0e477d9-b58a-4766-b341-ef28fa5d69ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:30.727746 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:30.727707    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:30.728366 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:30.727904    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:22:30.728366 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:30.727966    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret podName:9d037290-6471-4e48-bf2e-bcaa7de9cc64 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:32.727948433 +0000 UTC m=+12.377233413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret") pod "global-pull-secret-syncer-vcnp5" (UID: "9d037290-6471-4e48-bf2e-bcaa7de9cc64") : object "kube-system"/"original-pull-secret" not registered
May 25 11:22:30.931669 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:30.931599    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:30.931825 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:30.931721    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:30.931825 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:30.931734    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:30.931912 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:30.931848    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:30.931912 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:30.931894    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:30.932014 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:30.931958    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:32.741447 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:32.741404    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:32.742041 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:32.741562    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:22:32.742041 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:32.741650    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret podName:9d037290-6471-4e48-bf2e-bcaa7de9cc64 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:36.741627247 +0000 UTC m=+16.390912222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret") pod "global-pull-secret-syncer-vcnp5" (UID: "9d037290-6471-4e48-bf2e-bcaa7de9cc64") : object "kube-system"/"original-pull-secret" not registered
May 25 11:22:32.937788 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:32.937289    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:32.937788 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:32.937415    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:32.937788 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:32.937431    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:32.937788 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:32.937568    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:32.937788 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:32.937578    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:32.937788 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:32.937671    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:34.930270 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:34.930231    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:34.930270 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:34.930263    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:34.930761 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:34.930316    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:34.930761 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:34.930422    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:34.930761 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:34.930571    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:34.930761 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:34.930658    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:36.770856 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:36.770816    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:36.771296 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:36.770941    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:22:36.771296 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:36.771004    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret podName:9d037290-6471-4e48-bf2e-bcaa7de9cc64 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:44.770985916 +0000 UTC m=+24.420270891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret") pod "global-pull-secret-syncer-vcnp5" (UID: "9d037290-6471-4e48-bf2e-bcaa7de9cc64") : object "kube-system"/"original-pull-secret" not registered
May 25 11:22:36.930713 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:36.930472    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:36.930713 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:36.930508    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:36.930713 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:36.930549    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:36.930713 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:36.930661    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:36.930713 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:36.930714    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:36.931133 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:36.930813    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:38.584283 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:38.584245    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:38.584691 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:38.584411    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:38.584691 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:38.584493    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs podName:489b5117-49e0-45e5-b99a-3ad57febc838 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:54.584473224 +0000 UTC m=+34.233758196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs") pod "network-metrics-daemon-tpl95" (UID: "489b5117-49e0-45e5-b99a-3ad57febc838") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:38.685689 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:38.685647    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l7c\" (UniqueName: \"kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c\") pod \"network-check-target-9xqkb\" (UID: \"d0e477d9-b58a-4766-b341-ef28fa5d69ab\") " pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:38.685876 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:38.685810    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:22:38.685876 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:38.685833    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:22:38.685876 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:38.685846    2578 projected.go:194] Error preparing data for projected volume kube-api-access-z7l7c for pod openshift-network-diagnostics/network-check-target-9xqkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:38.686011 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:38.685906    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c podName:d0e477d9-b58a-4766-b341-ef28fa5d69ab nodeName:}" failed. No retries permitted until 2026-05-25 11:22:54.685889589 +0000 UTC m=+34.335174581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7l7c" (UniqueName: "kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c") pod "network-check-target-9xqkb" (UID: "d0e477d9-b58a-4766-b341-ef28fa5d69ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:38.930470 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:38.930356    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:38.930470 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:38.930372    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:38.930684 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:38.930529    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:38.930684 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:38.930591    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:38.930780 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:38.930740    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:38.930914 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:38.930867    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:40.931331 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:40.931298    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:40.931939 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:40.931355    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:40.931939 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:40.931440    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:40.931939 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:40.931469    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:40.931939 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:40.931552    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:40.931939 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:40.931653    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:42.931265 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:42.930879    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:42.931956 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:42.930879    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:42.931956 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:42.931350    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:42.931956 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:42.930880    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:42.931956 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:42.931400    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:42.931956 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:42.931488    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:43.003761 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.003710    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-29jf8" event={"ID":"c4ac3548-3f56-4896-8325-5a69d47b6008","Type":"ContainerStarted","Data":"2316807631f6e8ce90299fcfa017d8b1c893236af5d7ce6b4a7cb9b1938feb37"}
May 25 11:22:43.006363 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.006342    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:22:43.006673 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.006653    2578 generic.go:358] "Generic (PLEG): container finished" podID="1575503e-b475-4021-822e-f2e700cfbf03" containerID="f0329dd059da9419c2624d5782e1e1afd3c02a9b0456d725d8063c7d7f67ca83" exitCode=1
May 25 11:22:43.006765 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.006713    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r" event={"ID":"1575503e-b475-4021-822e-f2e700cfbf03","Type":"ContainerStarted","Data":"5ca95521e052bf633c4514b5203ef378d03ded3e7244e16ae403f535055da159"}
May 25 11:22:43.006765 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.006735    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r" event={"ID":"1575503e-b475-4021-822e-f2e700cfbf03","Type":"ContainerStarted","Data":"a581efd8fd84fddc0dc7f4a82f74291fe7c5da21ed1ac461a24c16fd31af5215"}
May 25 11:22:43.006765 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.006745    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r" event={"ID":"1575503e-b475-4021-822e-f2e700cfbf03","Type":"ContainerStarted","Data":"445920f1627a8e062695f651806c86a17908dff21f0acbf4216f610691177bae"}
May 25 11:22:43.006765 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.006753    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r" event={"ID":"1575503e-b475-4021-822e-f2e700cfbf03","Type":"ContainerStarted","Data":"aaf34855eda3fcd18b5b7388c9bf557a3d0e4f2a152c49d1dd811614dab3c221"}
May 25 11:22:43.006765 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.006761    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r" event={"ID":"1575503e-b475-4021-822e-f2e700cfbf03","Type":"ContainerDied","Data":"f0329dd059da9419c2624d5782e1e1afd3c02a9b0456d725d8063c7d7f67ca83"}
May 25 11:22:43.006968 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.006770    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r" event={"ID":"1575503e-b475-4021-822e-f2e700cfbf03","Type":"ContainerStarted","Data":"ff2babbc8c1e256c5103a60c16cd865e408d319cdc9a03c0cdf654f4bffc1ca4"}
May 25 11:22:43.008325 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.008095    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wkkdr" event={"ID":"6375e270-ce2d-4549-a415-343be34c4dfc","Type":"ContainerStarted","Data":"23b5ace184751ae7828391bd7f97388b089f0d630d889da8f6dd6e00ac0a4ac5"}
May 25 11:22:43.009505 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.009484    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42" event={"ID":"7f10ac15-85c6-4bca-8d2d-d65bc5d13456","Type":"ContainerStarted","Data":"d1dada7a4847fe6afaaa624a61c70d2354e0753ab47aeaa9b2e1a8510b0919f6"}
May 25 11:22:43.010836 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.010812    2578 generic.go:358] "Generic (PLEG): container finished" podID="62c640d4-74b5-4e3c-b772-f4e818c25bfd" containerID="b6ac6860bb8547f71adf217f09245360332580ca5150716bdf8b60a218d41e80" exitCode=0
May 25 11:22:43.010931 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.010889    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9jz7" event={"ID":"62c640d4-74b5-4e3c-b772-f4e818c25bfd","Type":"ContainerDied","Data":"b6ac6860bb8547f71adf217f09245360332580ca5150716bdf8b60a218d41e80"}
May 25 11:22:43.012283 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.012258    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-57jwf" event={"ID":"ac7c52f6-4dec-48a3-a106-0fb515461899","Type":"ContainerStarted","Data":"a021fd02aa0da8e1bdee520ad9c055aec510bd7901776bfe14261d35f18506f4"}
May 25 11:22:43.013556 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.013531    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wdc9d" event={"ID":"e93f1f05-2be2-471a-9f26-73d201d99aca","Type":"ContainerStarted","Data":"5f6e4f67fed6cf5a24c0bf17ee2dbba93f20dfe27a904accb84654d1cfe22643"}
May 25 11:22:43.014958 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.014933    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6xxd2" event={"ID":"7080cc52-e076-45dc-8653-d4e259dc912e","Type":"ContainerStarted","Data":"63cb4fb89c2203435e4dd347b4d4eaa9cd00137e52463c68cc8d35a58f904d7e"}
May 25 11:22:43.056716 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.056662    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6xxd2" podStartSLOduration=11.888789324 podStartE2EDuration="22.056647308s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:23.611334958 +0000 UTC m=+3.260619932" lastFinishedPulling="2026-05-25 11:22:33.779192929 +0000 UTC m=+13.428477916" observedRunningTime="2026-05-25 11:22:43.055863428 +0000 UTC m=+22.705148422" watchObservedRunningTime="2026-05-25 11:22:43.056647308 +0000 UTC m=+22.705932302"
May 25 11:22:43.056895 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.056842    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-29jf8" podStartSLOduration=3.7027508080000002 podStartE2EDuration="22.056834491s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:23.623846492 +0000 UTC m=+3.273131464" lastFinishedPulling="2026-05-25 11:22:41.977930166 +0000 UTC m=+21.627215147" observedRunningTime="2026-05-25 11:22:43.039386472 +0000 UTC m=+22.688671479" watchObservedRunningTime="2026-05-25 11:22:43.056834491 +0000 UTC m=+22.706119487"
May 25 11:22:43.111545 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.111489    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-wdc9d" podStartSLOduration=3.739006326 podStartE2EDuration="22.111475069s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:23.61856506 +0000 UTC m=+3.267850049" lastFinishedPulling="2026-05-25 11:22:41.991033819 +0000 UTC m=+21.640318792" observedRunningTime="2026-05-25 11:22:43.110699603 +0000 UTC m=+22.759984598" watchObservedRunningTime="2026-05-25 11:22:43.111475069 +0000 UTC m=+22.760760057"
May 25 11:22:43.138850 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.138800    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-wkkdr" podStartSLOduration=3.775605236 podStartE2EDuration="22.138785586s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:23.613236848 +0000 UTC m=+3.262521824" lastFinishedPulling="2026-05-25 11:22:41.976417184 +0000 UTC m=+21.625702174" observedRunningTime="2026-05-25 11:22:43.137984619 +0000 UTC m=+22.787269615" watchObservedRunningTime="2026-05-25 11:22:43.138785586 +0000 UTC m=+22.788070581"
May 25 11:22:43.201186 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.201100    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-57jwf" podStartSLOduration=3.844482501 podStartE2EDuration="22.201083867s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:23.620274218 +0000 UTC m=+3.269559195" lastFinishedPulling="2026-05-25 11:22:41.976875586 +0000 UTC m=+21.626160561" observedRunningTime="2026-05-25 11:22:43.200734495 +0000 UTC m=+22.850019491" watchObservedRunningTime="2026-05-25 11:22:43.201083867 +0000 UTC m=+22.850368896"
May 25 11:22:43.683134 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.683110    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 11:22:43.918733 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.918565    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T11:22:43.683126863Z","UUID":"7f877833-1abf-4232-815c-3f1af831ab91","Handler":null,"Name":"","Endpoint":""}
May 25 11:22:43.921764 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.921725    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 11:22:43.921764 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:43.921761    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 11:22:44.018578 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:44.018548    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-864ms" event={"ID":"95d5a6bc-e251-4a2e-a478-debb49fb507c","Type":"ContainerStarted","Data":"30bf94a35d64ce69275d29bddff566fcc3ab75f933a0c29c3e370c73694251bb"}
May 25 11:22:44.020167 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:44.020129    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42" event={"ID":"7f10ac15-85c6-4bca-8d2d-d65bc5d13456","Type":"ContainerStarted","Data":"3c1e5fb359b6322ff1eeb1fb54acf6b5d9e09dfc8de2fa2e68360c79a582b90e"}
May 25 11:22:44.036675 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:44.036629    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-864ms" podStartSLOduration=4.676491391 podStartE2EDuration="23.03661554s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:23.616397684 +0000 UTC m=+3.265682657" lastFinishedPulling="2026-05-25 11:22:41.976521833 +0000 UTC m=+21.625806806" observedRunningTime="2026-05-25 11:22:44.036138983 +0000 UTC m=+23.685423977" watchObservedRunningTime="2026-05-25 11:22:44.03661554 +0000 UTC m=+23.685900535"
May 25 11:22:44.832386 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:44.832322    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:44.832484 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:44.832423    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:22:44.832526 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:44.832485    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret podName:9d037290-6471-4e48-bf2e-bcaa7de9cc64 nodeName:}" failed. No retries permitted until 2026-05-25 11:23:00.832472835 +0000 UTC m=+40.481757808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret") pod "global-pull-secret-syncer-vcnp5" (UID: "9d037290-6471-4e48-bf2e-bcaa7de9cc64") : object "kube-system"/"original-pull-secret" not registered
May 25 11:22:44.872535 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:44.872501    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:44.873277 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:44.873256    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:44.931370 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:44.930702    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:44.931370 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:44.930725    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:44.931370 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:44.930843    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:44.931370 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:44.930870    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:44.931370 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:44.930999    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:44.931370 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:44.931079    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:45.024971 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:45.024942    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:22:45.025344 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:45.025255    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r" event={"ID":"1575503e-b475-4021-822e-f2e700cfbf03","Type":"ContainerStarted","Data":"ad7daf751180349fc425e26e7202042aa6ba7060808a12a8d33772303d5dbc05"}
May 25 11:22:45.027117 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:45.027090    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42" event={"ID":"7f10ac15-85c6-4bca-8d2d-d65bc5d13456","Type":"ContainerStarted","Data":"19a4c97c6d1641976766a44c9f4397b8ab3b4e6df5b3c2f3a93ca4e334e52946"}
May 25 11:22:45.065097 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:45.065045    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ngd42" podStartSLOduration=3.094453213 podStartE2EDuration="24.065029454s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:23.612150175 +0000 UTC m=+3.261435159" lastFinishedPulling="2026-05-25 11:22:44.582726428 +0000 UTC m=+24.232011400" observedRunningTime="2026-05-25 11:22:45.064559696 +0000 UTC m=+24.713844691" watchObservedRunningTime="2026-05-25 11:22:45.065029454 +0000 UTC m=+24.714314449"
May 25 11:22:45.237361 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:45.237276    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:45.237938 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:45.237917    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6xxd2"
May 25 11:22:46.930768 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:46.930728    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:46.931198 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:46.930787    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:46.931198 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:46.930902    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:46.931198 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:46.931016    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:46.931198 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:46.931061    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:46.931198 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:46.931126    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:48.930887 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:48.930713    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:48.931379 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:48.930711    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:48.931379 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:48.930973    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:48.931379 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:48.930771    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:48.931379 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:48.931024    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:48.931379 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:48.931094    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:49.036135 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.036097    2578 generic.go:358] "Generic (PLEG): container finished" podID="62c640d4-74b5-4e3c-b772-f4e818c25bfd" containerID="91676a02c47797ef79554c93bfab8d2353968847b392409ed15f195465d74f48" exitCode=0
May 25 11:22:49.036293 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.036180    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9jz7" event={"ID":"62c640d4-74b5-4e3c-b772-f4e818c25bfd","Type":"ContainerDied","Data":"91676a02c47797ef79554c93bfab8d2353968847b392409ed15f195465d74f48"}
May 25 11:22:49.039436 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.039419    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:22:49.039821 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.039771    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r" event={"ID":"1575503e-b475-4021-822e-f2e700cfbf03","Type":"ContainerStarted","Data":"0bc3101bd53bfabb3628ce64befc61747bf2c0dda418f212523199a4b1f6a274"}
May 25 11:22:49.040100 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.040069    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:49.040100 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.040099    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:49.040258 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.040111    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:49.040258 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.040227    2578 scope.go:117] "RemoveContainer" containerID="f0329dd059da9419c2624d5782e1e1afd3c02a9b0456d725d8063c7d7f67ca83"
May 25 11:22:49.057599 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.057577    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:49.057699 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.057654    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:22:49.980601 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.980353    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tpl95"]
May 25 11:22:49.981006 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.980696    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:49.981006 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:49.980791    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:49.989973 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.989905    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vcnp5"]
May 25 11:22:49.990083 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.990000    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:49.990115 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:49.990077    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:49.993671 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.993647    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9xqkb"]
May 25 11:22:49.993796 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:49.993725    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:49.993856 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:49.993817    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:50.044706 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:50.044681    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:22:50.045037 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:50.045015    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r" event={"ID":"1575503e-b475-4021-822e-f2e700cfbf03","Type":"ContainerStarted","Data":"26508609fbd2d603c1e80a3bbb7626b95c6add8251bc8b4b2b1894b4fb3be101"}
May 25 11:22:50.047066 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:50.047044    2578 generic.go:358] "Generic (PLEG): container finished" podID="62c640d4-74b5-4e3c-b772-f4e818c25bfd" containerID="83e60d5951ed2a2221bd3cd99c8b4ccfc26544484d3d10f1ce47a515a68454a4" exitCode=0
May 25 11:22:50.047157 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:50.047079    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9jz7" event={"ID":"62c640d4-74b5-4e3c-b772-f4e818c25bfd","Type":"ContainerDied","Data":"83e60d5951ed2a2221bd3cd99c8b4ccfc26544484d3d10f1ce47a515a68454a4"}
May 25 11:22:50.095608 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:50.095559    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r" podStartSLOduration=10.682405662 podStartE2EDuration="29.095545608s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:23.617211996 +0000 UTC m=+3.266496975" lastFinishedPulling="2026-05-25 11:22:42.030351945 +0000 UTC m=+21.679636921" observedRunningTime="2026-05-25 11:22:50.087143544 +0000 UTC m=+29.736428541" watchObservedRunningTime="2026-05-25 11:22:50.095545608 +0000 UTC m=+29.744830596"
May 25 11:22:51.051258 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:51.051224    2578 generic.go:358] "Generic (PLEG): container finished" podID="62c640d4-74b5-4e3c-b772-f4e818c25bfd" containerID="7026a69febdff7c4d68b6eff64688aa8fa856972357aa4e90ee6705f6600882d" exitCode=0
May 25 11:22:51.051895 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:51.051307    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9jz7" event={"ID":"62c640d4-74b5-4e3c-b772-f4e818c25bfd","Type":"ContainerDied","Data":"7026a69febdff7c4d68b6eff64688aa8fa856972357aa4e90ee6705f6600882d"}
May 25 11:22:51.930496 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:51.930393    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:51.930496 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:51.930447    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:51.930711 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:51.930538    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:51.930711 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:51.930588    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:51.930711 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:51.930695    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:51.930828 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:51.930762    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:53.930474 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:53.930421    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:53.930474 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:53.930480    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:53.931143 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:53.930559    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9xqkb" podUID="d0e477d9-b58a-4766-b341-ef28fa5d69ab"
May 25 11:22:53.931143 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:53.930628    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vcnp5" podUID="9d037290-6471-4e48-bf2e-bcaa7de9cc64"
May 25 11:22:53.931143 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:53.930643    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:53.931143 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:53.930739    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:22:54.595010 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.594988    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-58.ec2.internal" event="NodeReady"
May 25 11:22:54.595230 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.595114    2578 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 11:22:54.605258 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.605237    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:54.605342 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.605332    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:54.605386 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.605378    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs podName:489b5117-49e0-45e5-b99a-3ad57febc838 nodeName:}" failed. No retries permitted until 2026-05-25 11:23:26.605364834 +0000 UTC m=+66.254649807 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs") pod "network-metrics-daemon-tpl95" (UID: "489b5117-49e0-45e5-b99a-3ad57febc838") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:22:54.644964 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.644929    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"]
May 25 11:22:54.648025 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.648010    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:22:54.651895 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.651686    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 25 11:22:54.651895 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.651686    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 25 11:22:54.651895 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.651724    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 25 11:22:54.651895 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.651744    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 25 11:22:54.655674 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.654135    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"]
May 25 11:22:54.658028 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.657757    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m"]
May 25 11:22:54.658282 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.658259    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.661192 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.661149    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 25 11:22:54.662591 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.661782    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 25 11:22:54.662591 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.662006    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 25 11:22:54.662842 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.662824    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 25 11:22:54.663590 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.663571    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-756c5b7b96-tnzsx"]
May 25 11:22:54.663751 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.663732    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m"
May 25 11:22:54.666531 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.666503    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"]
May 25 11:22:54.666678 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.666634    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.667139 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.667122    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 25 11:22:54.667565 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.667140    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-v8csk\""
May 25 11:22:54.669643 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.669618    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 11:22:54.669774 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.669753    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 11:22:54.669888 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.669844    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w59mk\""
May 25 11:22:54.670087 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.669966    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 11:22:54.675136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.675114    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"]
May 25 11:22:54.677086 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.677064    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 11:22:54.681834 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.678914    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-756c5b7b96-tnzsx"]
May 25 11:22:54.681834 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.680529    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rlr4p"]
May 25 11:22:54.683449 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.683423    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h5vl5"]
May 25 11:22:54.683615 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.683597    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.686153 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.686135    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 11:22:54.686841 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.686824    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m"]
May 25 11:22:54.686928 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.686919    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:22:54.686981 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.686969    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 11:22:54.687194 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.687181    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mbq6k\""
May 25 11:22:54.690149 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.690129    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 11:22:54.690593 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.690576    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 11:22:54.690788 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.690772    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n7v95\""
May 25 11:22:54.690949 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.690932    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 11:22:54.695916 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.695895    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rlr4p"]
May 25 11:22:54.698288 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.698267    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h5vl5"]
May 25 11:22:54.705797 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.705771    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l7c\" (UniqueName: \"kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c\") pod \"network-check-target-9xqkb\" (UID: \"d0e477d9-b58a-4766-b341-ef28fa5d69ab\") " pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:54.705971 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.705953    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:22:54.706044 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.705979    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:22:54.706044 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.705994    2578 projected.go:194] Error preparing data for projected volume kube-api-access-z7l7c for pod openshift-network-diagnostics/network-check-target-9xqkb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:54.706112 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.706055    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c podName:d0e477d9-b58a-4766-b341-ef28fa5d69ab nodeName:}" failed. No retries permitted until 2026-05-25 11:23:26.706038837 +0000 UTC m=+66.355323823 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7l7c" (UniqueName: "kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c") pod "network-check-target-9xqkb" (UID: "d0e477d9-b58a-4766-b341-ef28fa5d69ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:22:54.806136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-installation-pull-secrets\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.806323 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vlrb\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-kube-api-access-4vlrb\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.806323 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70bb9cc9-7bb3-443b-823d-432d43967d4b-tmp\") pod \"klusterlet-addon-workmgr-598b6b76dc-hbkwj\" (UID: \"70bb9cc9-7bb3-443b-823d-432d43967d4b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:22:54.806323 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpckd\" (UniqueName: \"kubernetes.io/projected/70bb9cc9-7bb3-443b-823d-432d43967d4b-kube-api-access-zpckd\") pod \"klusterlet-addon-workmgr-598b6b76dc-hbkwj\" (UID: \"70bb9cc9-7bb3-443b-823d-432d43967d4b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:22:54.806323 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-ca-trust-extracted\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.806323 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ec1becf5-75b4-480e-8f33-f131f7186bf5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bcd655556-l4w8m\" (UID: \"ec1becf5-75b4-480e-8f33-f131f7186bf5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m"
May 25 11:22:54.806559 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-ca\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.806559 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-hub\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.806559 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvsp\" (UniqueName: \"kubernetes.io/projected/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-kube-api-access-2mvsp\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:22:54.806559 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/38fd9e26-33a6-4a32-b382-de9a5433b29e-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.806559 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/70bb9cc9-7bb3-443b-823d-432d43967d4b-klusterlet-config\") pod \"klusterlet-addon-workmgr-598b6b76dc-hbkwj\" (UID: \"70bb9cc9-7bb3-443b-823d-432d43967d4b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:22:54.806766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.806766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-tmp-dir\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.806766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6nfv\" (UniqueName: \"kubernetes.io/projected/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-kube-api-access-z6nfv\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.806766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-image-registry-private-configuration\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.806766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-config-volume\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.806766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.806766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sltt5\" (UniqueName: \"kubernetes.io/projected/38fd9e26-33a6-4a32-b382-de9a5433b29e-kube-api-access-sltt5\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.806766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-trusted-ca\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.807114 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-bound-sa-token\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.807114 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-certificates\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.807114 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncb2m\" (UniqueName: \"kubernetes.io/projected/ec1becf5-75b4-480e-8f33-f131f7186bf5-kube-api-access-ncb2m\") pod \"managed-serviceaccount-addon-agent-6bcd655556-l4w8m\" (UID: \"ec1becf5-75b4-480e-8f33-f131f7186bf5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m"
May 25 11:22:54.807114 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.807114 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.807114 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.806938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:22:54.908286 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908198    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/70bb9cc9-7bb3-443b-823d-432d43967d4b-klusterlet-config\") pod \"klusterlet-addon-workmgr-598b6b76dc-hbkwj\" (UID: \"70bb9cc9-7bb3-443b-823d-432d43967d4b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:22:54.908286 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908237    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.908286 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908264    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-tmp-dir\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.908286 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908286    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z6nfv\" (UniqueName: \"kubernetes.io/projected/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-kube-api-access-z6nfv\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908309    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-image-registry-private-configuration\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908340    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-config-volume\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908389    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908416    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sltt5\" (UniqueName: \"kubernetes.io/projected/38fd9e26-33a6-4a32-b382-de9a5433b29e-kube-api-access-sltt5\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908443    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-trusted-ca\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908491    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-bound-sa-token\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908520    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-certificates\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908546    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ncb2m\" (UniqueName: \"kubernetes.io/projected/ec1becf5-75b4-480e-8f33-f131f7186bf5-kube-api-access-ncb2m\") pod \"managed-serviceaccount-addon-agent-6bcd655556-l4w8m\" (UID: \"ec1becf5-75b4-480e-8f33-f131f7186bf5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908565    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908583    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908602    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:22:54.908646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908640    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-installation-pull-secrets\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908664    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4vlrb\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-kube-api-access-4vlrb\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908686    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70bb9cc9-7bb3-443b-823d-432d43967d4b-tmp\") pod \"klusterlet-addon-workmgr-598b6b76dc-hbkwj\" (UID: \"70bb9cc9-7bb3-443b-823d-432d43967d4b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908702    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zpckd\" (UniqueName: \"kubernetes.io/projected/70bb9cc9-7bb3-443b-823d-432d43967d4b-kube-api-access-zpckd\") pod \"klusterlet-addon-workmgr-598b6b76dc-hbkwj\" (UID: \"70bb9cc9-7bb3-443b-823d-432d43967d4b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908723    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-ca-trust-extracted\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908731    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-tmp-dir\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908740    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ec1becf5-75b4-480e-8f33-f131f7186bf5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bcd655556-l4w8m\" (UID: \"ec1becf5-75b4-480e-8f33-f131f7186bf5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m"
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908823    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-ca\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908850    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-hub\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908895    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvsp\" (UniqueName: \"kubernetes.io/projected/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-kube-api-access-2mvsp\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.908897    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.908938    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/38fd9e26-33a6-4a32-b382-de9a5433b29e-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.909136 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.908980    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert podName:f6151b3e-2de0-49db-a40a-3a4d0e75c3e1 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:55.408956962 +0000 UTC m=+35.058241948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert") pod "ingress-canary-h5vl5" (UID: "f6151b3e-2de0-49db-a40a-3a4d0e75c3e1") : secret "canary-serving-cert" not found
May 25 11:22:54.909627 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.909548    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-trusted-ca\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.909627 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.909556    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-config-volume\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.909627 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.909573    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/38fd9e26-33a6-4a32-b382-de9a5433b29e-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.909750 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.909697    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:22:54.909750 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.909709    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-756c5b7b96-tnzsx: secret "image-registry-tls" not found
May 25 11:22:54.909831 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.909777    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls podName:7d43b17c-c5b2-419a-b6a2-06e83911b9ca nodeName:}" failed. No retries permitted until 2026-05-25 11:22:55.409762686 +0000 UTC m=+35.059047677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls") pod "image-registry-756c5b7b96-tnzsx" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca") : secret "image-registry-tls" not found
May 25 11:22:54.909879 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.909830    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:22:54.909879 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:54.909856    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls podName:f0720f3e-ce4d-4fc2-a5de-5b624520ae3c nodeName:}" failed. No retries permitted until 2026-05-25 11:22:55.409847783 +0000 UTC m=+35.059132799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls") pod "dns-default-rlr4p" (UID: "f0720f3e-ce4d-4fc2-a5de-5b624520ae3c") : secret "dns-default-metrics-tls" not found
May 25 11:22:54.909965 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.909900    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-certificates\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.910345 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.910234    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70bb9cc9-7bb3-443b-823d-432d43967d4b-tmp\") pod \"klusterlet-addon-workmgr-598b6b76dc-hbkwj\" (UID: \"70bb9cc9-7bb3-443b-823d-432d43967d4b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:22:54.911528 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.911503    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-ca-trust-extracted\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.913517 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.913491    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-ca\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.913517 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.913506    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-installation-pull-secrets\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.913699 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.913592    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/70bb9cc9-7bb3-443b-823d-432d43967d4b-klusterlet-config\") pod \"klusterlet-addon-workmgr-598b6b76dc-hbkwj\" (UID: \"70bb9cc9-7bb3-443b-823d-432d43967d4b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:22:54.914063 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.914040    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-image-registry-private-configuration\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.914163 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.914146    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.915806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.915780    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.916377 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.916356    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ec1becf5-75b4-480e-8f33-f131f7186bf5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bcd655556-l4w8m\" (UID: \"ec1becf5-75b4-480e-8f33-f131f7186bf5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m"
May 25 11:22:54.916530 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.916513    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/38fd9e26-33a6-4a32-b382-de9a5433b29e-hub\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.920295 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.920221    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6nfv\" (UniqueName: \"kubernetes.io/projected/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-kube-api-access-z6nfv\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:54.925796 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.925769    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sltt5\" (UniqueName: \"kubernetes.io/projected/38fd9e26-33a6-4a32-b382-de9a5433b29e-kube-api-access-sltt5\") pod \"cluster-proxy-proxy-agent-77654fcb55-cv7br\" (UID: \"38fd9e26-33a6-4a32-b382-de9a5433b29e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.926301 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.926029    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvsp\" (UniqueName: \"kubernetes.io/projected/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-kube-api-access-2mvsp\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:22:54.926301 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.926044    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpckd\" (UniqueName: \"kubernetes.io/projected/70bb9cc9-7bb3-443b-823d-432d43967d4b-kube-api-access-zpckd\") pod \"klusterlet-addon-workmgr-598b6b76dc-hbkwj\" (UID: \"70bb9cc9-7bb3-443b-823d-432d43967d4b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:22:54.926715 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.926693    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncb2m\" (UniqueName: \"kubernetes.io/projected/ec1becf5-75b4-480e-8f33-f131f7186bf5-kube-api-access-ncb2m\") pod \"managed-serviceaccount-addon-agent-6bcd655556-l4w8m\" (UID: \"ec1becf5-75b4-480e-8f33-f131f7186bf5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m"
May 25 11:22:54.927745 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.927699    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vlrb\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-kube-api-access-4vlrb\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.927970 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.927950    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-bound-sa-token\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:54.962765 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.962733    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:22:54.981752 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.981720    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:22:54.991043 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:54.991013    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m"
May 25 11:22:55.108627 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.108596    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"]
May 25 11:22:55.414567 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.414533    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:55.414757 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.414577    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:55.414757 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.414596    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:22:55.414757 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:55.414699    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:22:55.414912 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:55.414783    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls podName:f0720f3e-ce4d-4fc2-a5de-5b624520ae3c nodeName:}" failed. No retries permitted until 2026-05-25 11:22:56.414759107 +0000 UTC m=+36.064044082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls") pod "dns-default-rlr4p" (UID: "f0720f3e-ce4d-4fc2-a5de-5b624520ae3c") : secret "dns-default-metrics-tls" not found
May 25 11:22:55.414912 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:55.414703    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:22:55.414912 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:55.414853    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert podName:f6151b3e-2de0-49db-a40a-3a4d0e75c3e1 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:56.414838881 +0000 UTC m=+36.064123858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert") pod "ingress-canary-h5vl5" (UID: "f6151b3e-2de0-49db-a40a-3a4d0e75c3e1") : secret "canary-serving-cert" not found
May 25 11:22:55.414912 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:55.414707    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:22:55.414912 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:55.414871    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-756c5b7b96-tnzsx: secret "image-registry-tls" not found
May 25 11:22:55.414912 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:55.414903    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls podName:7d43b17c-c5b2-419a-b6a2-06e83911b9ca nodeName:}" failed. No retries permitted until 2026-05-25 11:22:56.414892573 +0000 UTC m=+36.064177577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls") pod "image-registry-756c5b7b96-tnzsx" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca") : secret "image-registry-tls" not found
May 25 11:22:55.931601 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.930656    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:22:55.931601 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.930679    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:22:55.931601 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.931024    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:22:55.934075 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.934048    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:22:55.935807 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.935202    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:22:55.935807 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.935218    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x4v4p\""
May 25 11:22:55.935807 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.935280    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:22:55.935807 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.935298    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 11:22:55.935807 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:55.935447    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-brg2t\""
May 25 11:22:56.424008 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:56.423913    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:56.424008 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:56.423966    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:56.424008 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:56.423994    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:22:56.424498 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:56.424082    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:22:56.424498 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:56.424115    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:22:56.424498 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:56.424117    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:22:56.424498 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:56.424140    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-756c5b7b96-tnzsx: secret "image-registry-tls" not found
May 25 11:22:56.424498 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:56.424183    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls podName:f0720f3e-ce4d-4fc2-a5de-5b624520ae3c nodeName:}" failed. No retries permitted until 2026-05-25 11:22:58.424149127 +0000 UTC m=+38.073434117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls") pod "dns-default-rlr4p" (UID: "f0720f3e-ce4d-4fc2-a5de-5b624520ae3c") : secret "dns-default-metrics-tls" not found
May 25 11:22:56.424498 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:56.424205    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert podName:f6151b3e-2de0-49db-a40a-3a4d0e75c3e1 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:58.424193621 +0000 UTC m=+38.073478594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert") pod "ingress-canary-h5vl5" (UID: "f6151b3e-2de0-49db-a40a-3a4d0e75c3e1") : secret "canary-serving-cert" not found
May 25 11:22:56.424498 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:56.424220    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls podName:7d43b17c-c5b2-419a-b6a2-06e83911b9ca nodeName:}" failed. No retries permitted until 2026-05-25 11:22:58.424211359 +0000 UTC m=+38.073496333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls") pod "image-registry-756c5b7b96-tnzsx" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca") : secret "image-registry-tls" not found
May 25 11:22:57.153145 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:57.153114    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bb9cc9_7bb3_443b_823d_432d43967d4b.slice/crio-cd2a3191990c7300825ff49be4d3f7d7feea5e99739d33482fde6d82b117d0cc WatchSource:0}: Error finding container cd2a3191990c7300825ff49be4d3f7d7feea5e99739d33482fde6d82b117d0cc: Status 404 returned error can't find the container with id cd2a3191990c7300825ff49be4d3f7d7feea5e99739d33482fde6d82b117d0cc
May 25 11:22:57.302319 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:57.302287    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"]
May 25 11:22:57.306255 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:57.306228    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m"]
May 25 11:22:57.326397 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:57.326368    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38fd9e26_33a6_4a32_b382_de9a5433b29e.slice/crio-3788af3946e6d13d501ed1a403ce64a7059ca624837fb4a3ba7bbf7067ddf47d WatchSource:0}: Error finding container 3788af3946e6d13d501ed1a403ce64a7059ca624837fb4a3ba7bbf7067ddf47d: Status 404 returned error can't find the container with id 3788af3946e6d13d501ed1a403ce64a7059ca624837fb4a3ba7bbf7067ddf47d
May 25 11:22:57.326621 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:22:57.326605    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec1becf5_75b4_480e_8f33_f131f7186bf5.slice/crio-4aa190c0527cfe39a155497e4e4e5449f7fa0e956ce53fad51ae72e471b74fb4 WatchSource:0}: Error finding container 4aa190c0527cfe39a155497e4e4e5449f7fa0e956ce53fad51ae72e471b74fb4: Status 404 returned error can't find the container with id 4aa190c0527cfe39a155497e4e4e5449f7fa0e956ce53fad51ae72e471b74fb4
May 25 11:22:58.072627 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:58.072396    2578 generic.go:358] "Generic (PLEG): container finished" podID="62c640d4-74b5-4e3c-b772-f4e818c25bfd" containerID="6b0cd5767d562c8a9c88d0aded66ef5bba0a153b80baaabfc497bcf260ea7d12" exitCode=0
May 25 11:22:58.073159 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:58.072682    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9jz7" event={"ID":"62c640d4-74b5-4e3c-b772-f4e818c25bfd","Type":"ContainerDied","Data":"6b0cd5767d562c8a9c88d0aded66ef5bba0a153b80baaabfc497bcf260ea7d12"}
May 25 11:22:58.073988 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:58.073949    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" event={"ID":"38fd9e26-33a6-4a32-b382-de9a5433b29e","Type":"ContainerStarted","Data":"3788af3946e6d13d501ed1a403ce64a7059ca624837fb4a3ba7bbf7067ddf47d"}
May 25 11:22:58.075677 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:58.075626    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m" event={"ID":"ec1becf5-75b4-480e-8f33-f131f7186bf5","Type":"ContainerStarted","Data":"4aa190c0527cfe39a155497e4e4e5449f7fa0e956ce53fad51ae72e471b74fb4"}
May 25 11:22:58.080139 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:58.080063    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj" event={"ID":"70bb9cc9-7bb3-443b-823d-432d43967d4b","Type":"ContainerStarted","Data":"cd2a3191990c7300825ff49be4d3f7d7feea5e99739d33482fde6d82b117d0cc"}
May 25 11:22:58.443712 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:58.443617    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:22:58.443712 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:58.443677    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:22:58.443712 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:58.443704    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:22:58.443974 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:58.443907    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:22:58.443974 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:58.443969    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert podName:f6151b3e-2de0-49db-a40a-3a4d0e75c3e1 nodeName:}" failed. No retries permitted until 2026-05-25 11:23:02.443951562 +0000 UTC m=+42.093236542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert") pod "ingress-canary-h5vl5" (UID: "f6151b3e-2de0-49db-a40a-3a4d0e75c3e1") : secret "canary-serving-cert" not found
May 25 11:22:58.444391 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:58.444369    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:22:58.444488 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:58.444433    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls podName:f0720f3e-ce4d-4fc2-a5de-5b624520ae3c nodeName:}" failed. No retries permitted until 2026-05-25 11:23:02.444416606 +0000 UTC m=+42.093701584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls") pod "dns-default-rlr4p" (UID: "f0720f3e-ce4d-4fc2-a5de-5b624520ae3c") : secret "dns-default-metrics-tls" not found
May 25 11:22:58.444550 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:58.444505    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:22:58.444550 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:58.444514    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-756c5b7b96-tnzsx: secret "image-registry-tls" not found
May 25 11:22:58.444550 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:22:58.444540    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls podName:7d43b17c-c5b2-419a-b6a2-06e83911b9ca nodeName:}" failed. No retries permitted until 2026-05-25 11:23:02.444532681 +0000 UTC m=+42.093817653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls") pod "image-registry-756c5b7b96-tnzsx" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca") : secret "image-registry-tls" not found
May 25 11:22:59.089129 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:59.088256    2578 generic.go:358] "Generic (PLEG): container finished" podID="62c640d4-74b5-4e3c-b772-f4e818c25bfd" containerID="e223b8cb58fdd0e3bb7222d5a6a054804ca34234447e0cfb9165107f93b94700" exitCode=0
May 25 11:22:59.089129 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:22:59.088312    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9jz7" event={"ID":"62c640d4-74b5-4e3c-b772-f4e818c25bfd","Type":"ContainerDied","Data":"e223b8cb58fdd0e3bb7222d5a6a054804ca34234447e0cfb9165107f93b94700"}
May 25 11:23:00.868039 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:00.867994    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:23:00.872287 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:00.872255    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d037290-6471-4e48-bf2e-bcaa7de9cc64-original-pull-secret\") pod \"global-pull-secret-syncer-vcnp5\" (UID: \"9d037290-6471-4e48-bf2e-bcaa7de9cc64\") " pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:23:01.056862 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:01.056817    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vcnp5"
May 25 11:23:02.482610 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:02.482562    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:23:02.483058 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:02.482619    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:23:02.483058 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:02.482639    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:23:02.483058 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:02.482740    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:23:02.483058 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:02.482761    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:23:02.483058 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:02.482787    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:23:02.483058 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:02.482804    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-756c5b7b96-tnzsx: secret "image-registry-tls" not found
May 25 11:23:02.483058 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:02.482841    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls podName:f0720f3e-ce4d-4fc2-a5de-5b624520ae3c nodeName:}" failed. No retries permitted until 2026-05-25 11:23:10.482819859 +0000 UTC m=+50.132104833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls") pod "dns-default-rlr4p" (UID: "f0720f3e-ce4d-4fc2-a5de-5b624520ae3c") : secret "dns-default-metrics-tls" not found
May 25 11:23:02.483058 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:02.482863    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert podName:f6151b3e-2de0-49db-a40a-3a4d0e75c3e1 nodeName:}" failed. No retries permitted until 2026-05-25 11:23:10.482854396 +0000 UTC m=+50.132139374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert") pod "ingress-canary-h5vl5" (UID: "f6151b3e-2de0-49db-a40a-3a4d0e75c3e1") : secret "canary-serving-cert" not found
May 25 11:23:02.483058 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:02.482873    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls podName:7d43b17c-c5b2-419a-b6a2-06e83911b9ca nodeName:}" failed. No retries permitted until 2026-05-25 11:23:10.482867784 +0000 UTC m=+50.132152757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls") pod "image-registry-756c5b7b96-tnzsx" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca") : secret "image-registry-tls" not found
May 25 11:23:04.975034 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:04.974970    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vcnp5"]
May 25 11:23:04.977752 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:23:04.977724    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d037290_6471_4e48_bf2e_bcaa7de9cc64.slice/crio-b387a902764d47d405ff317bcf016967c66204255d9601ee25e705fa74535745 WatchSource:0}: Error finding container b387a902764d47d405ff317bcf016967c66204255d9601ee25e705fa74535745: Status 404 returned error can't find the container with id b387a902764d47d405ff317bcf016967c66204255d9601ee25e705fa74535745
May 25 11:23:05.102866 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:05.102783    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l9jz7" event={"ID":"62c640d4-74b5-4e3c-b772-f4e818c25bfd","Type":"ContainerStarted","Data":"d8316a192f416747bde24b9b33b3d1a4ea1a47b340dedcd1239605dcad216a75"}
May 25 11:23:05.103990 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:05.103956    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj" event={"ID":"70bb9cc9-7bb3-443b-823d-432d43967d4b","Type":"ContainerStarted","Data":"8a4c08fa915b3d7fe326a30b3998b6dbff3f0b87fb87beecdebe2c657ddf83fa"}
May 25 11:23:05.104172 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:05.104151    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:23:05.105101 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:05.105075    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vcnp5" event={"ID":"9d037290-6471-4e48-bf2e-bcaa7de9cc64","Type":"ContainerStarted","Data":"b387a902764d47d405ff317bcf016967c66204255d9601ee25e705fa74535745"}
May 25 11:23:05.106029 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:05.106005    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:23:05.106379 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:05.106359    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" event={"ID":"38fd9e26-33a6-4a32-b382-de9a5433b29e","Type":"ContainerStarted","Data":"f42d512992e236f440c31251dac4bcf3594e2f70763650a1f0ccae1eed0dc6c8"}
May 25 11:23:05.107626 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:05.107609    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m" event={"ID":"ec1becf5-75b4-480e-8f33-f131f7186bf5","Type":"ContainerStarted","Data":"54ab2ec7779c490aa960dde1ebd61bf6349bc7db71ccf0996994f214800388f6"}
May 25 11:23:05.131649 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:05.131215    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l9jz7" podStartSLOduration=10.393012894 podStartE2EDuration="44.131195853s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:23.623618664 +0000 UTC m=+3.272903652" lastFinishedPulling="2026-05-25 11:22:57.36180163 +0000 UTC m=+37.011086611" observedRunningTime="2026-05-25 11:23:05.128933214 +0000 UTC m=+44.778218210" watchObservedRunningTime="2026-05-25 11:23:05.131195853 +0000 UTC m=+44.780480849"
May 25 11:23:05.149352 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:05.149307    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m" podStartSLOduration=36.655056659 podStartE2EDuration="44.149294012s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:57.338900713 +0000 UTC m=+36.988185687" lastFinishedPulling="2026-05-25 11:23:04.833138052 +0000 UTC m=+44.482423040" observedRunningTime="2026-05-25 11:23:05.148371742 +0000 UTC m=+44.797656736" watchObservedRunningTime="2026-05-25 11:23:05.149294012 +0000 UTC m=+44.798579011"
May 25 11:23:05.168987 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:05.168943    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj" podStartSLOduration=36.490084819 podStartE2EDuration="44.168931228s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:57.155403608 +0000 UTC m=+36.804688585" lastFinishedPulling="2026-05-25 11:23:04.834250013 +0000 UTC m=+44.483534994" observedRunningTime="2026-05-25 11:23:05.16853193 +0000 UTC m=+44.817816924" watchObservedRunningTime="2026-05-25 11:23:05.168931228 +0000 UTC m=+44.818216225"
May 25 11:23:09.117058 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:09.117030    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" event={"ID":"38fd9e26-33a6-4a32-b382-de9a5433b29e","Type":"ContainerStarted","Data":"8b2ac88340a260eee4815efaab56daa66c76c607afa279c8644845fb2eed0315"}
May 25 11:23:10.125027 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:10.122498    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" event={"ID":"38fd9e26-33a6-4a32-b382-de9a5433b29e","Type":"ContainerStarted","Data":"6e810f1b93e895d832506f8679e3cea5828a27dcea51dc11c271ff54566a44d7"}
May 25 11:23:10.126485 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:10.126439    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vcnp5" event={"ID":"9d037290-6471-4e48-bf2e-bcaa7de9cc64","Type":"ContainerStarted","Data":"0e0b53f89a7c4c518cdb2720dc09de2202877dac29bca324a6c2830d37b4a33a"}
May 25 11:23:10.144026 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:10.143978    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" podStartSLOduration=37.485509742 podStartE2EDuration="49.143965021s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:22:57.3387986 +0000 UTC m=+36.988083573" lastFinishedPulling="2026-05-25 11:23:08.99725388 +0000 UTC m=+48.646538852" observedRunningTime="2026-05-25 11:23:10.142651111 +0000 UTC m=+49.791936106" watchObservedRunningTime="2026-05-25 11:23:10.143965021 +0000 UTC m=+49.793250009"
May 25 11:23:10.162990 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:10.162933    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vcnp5" podStartSLOduration=36.822850503 podStartE2EDuration="41.162918591s" podCreationTimestamp="2026-05-25 11:22:29 +0000 UTC" firstStartedPulling="2026-05-25 11:23:04.979718671 +0000 UTC m=+44.629003644" lastFinishedPulling="2026-05-25 11:23:09.319786742 +0000 UTC m=+48.969071732" observedRunningTime="2026-05-25 11:23:10.162002893 +0000 UTC m=+49.811287888" watchObservedRunningTime="2026-05-25 11:23:10.162918591 +0000 UTC m=+49.812203585"
May 25 11:23:10.540030 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:10.539991    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:23:10.540030 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:10.540033    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:23:10.540242 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:10.540051    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:23:10.540242 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:10.540137    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:23:10.540242 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:10.540140    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:23:10.540242 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:10.540183    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert podName:f6151b3e-2de0-49db-a40a-3a4d0e75c3e1 nodeName:}" failed. No retries permitted until 2026-05-25 11:23:26.540170616 +0000 UTC m=+66.189455589 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert") pod "ingress-canary-h5vl5" (UID: "f6151b3e-2de0-49db-a40a-3a4d0e75c3e1") : secret "canary-serving-cert" not found
May 25 11:23:10.540242 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:10.540205    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls podName:f0720f3e-ce4d-4fc2-a5de-5b624520ae3c nodeName:}" failed. No retries permitted until 2026-05-25 11:23:26.540191781 +0000 UTC m=+66.189476753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls") pod "dns-default-rlr4p" (UID: "f0720f3e-ce4d-4fc2-a5de-5b624520ae3c") : secret "dns-default-metrics-tls" not found
May 25 11:23:10.540242 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:10.540143    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:23:10.540242 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:10.540220    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-756c5b7b96-tnzsx: secret "image-registry-tls" not found
May 25 11:23:10.540242 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:10.540242    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls podName:7d43b17c-c5b2-419a-b6a2-06e83911b9ca nodeName:}" failed. No retries permitted until 2026-05-25 11:23:26.540236053 +0000 UTC m=+66.189521027 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls") pod "image-registry-756c5b7b96-tnzsx" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca") : secret "image-registry-tls" not found
May 25 11:23:21.063285 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:21.063246    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pvk9r"
May 25 11:23:26.558199 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.558156    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:23:26.558199 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.558205    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:23:26.558747 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.558223    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:23:26.558747 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:26.558305    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:23:26.558747 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:26.558308    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:23:26.558747 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:26.558320    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:23:26.558747 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:26.558336    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-756c5b7b96-tnzsx: secret "image-registry-tls" not found
May 25 11:23:26.558747 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:26.558372    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls podName:f0720f3e-ce4d-4fc2-a5de-5b624520ae3c nodeName:}" failed. No retries permitted until 2026-05-25 11:23:58.558353995 +0000 UTC m=+98.207638969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls") pod "dns-default-rlr4p" (UID: "f0720f3e-ce4d-4fc2-a5de-5b624520ae3c") : secret "dns-default-metrics-tls" not found
May 25 11:23:26.558747 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:26.558395    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert podName:f6151b3e-2de0-49db-a40a-3a4d0e75c3e1 nodeName:}" failed. No retries permitted until 2026-05-25 11:23:58.558388763 +0000 UTC m=+98.207673735 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert") pod "ingress-canary-h5vl5" (UID: "f6151b3e-2de0-49db-a40a-3a4d0e75c3e1") : secret "canary-serving-cert" not found
May 25 11:23:26.558747 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:26.558406    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls podName:7d43b17c-c5b2-419a-b6a2-06e83911b9ca nodeName:}" failed. No retries permitted until 2026-05-25 11:23:58.55840113 +0000 UTC m=+98.207686103 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls") pod "image-registry-756c5b7b96-tnzsx" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca") : secret "image-registry-tls" not found
May 25 11:23:26.659285 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.659242    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:23:26.664998 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.664977    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:23:26.669497 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:26.669483    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 11:23:26.669549 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:26.669535    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs podName:489b5117-49e0-45e5-b99a-3ad57febc838 nodeName:}" failed. No retries permitted until 2026-05-25 11:24:30.669521264 +0000 UTC m=+130.318806237 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs") pod "network-metrics-daemon-tpl95" (UID: "489b5117-49e0-45e5-b99a-3ad57febc838") : secret "metrics-daemon-secret" not found
May 25 11:23:26.760263 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.760224    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l7c\" (UniqueName: \"kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c\") pod \"network-check-target-9xqkb\" (UID: \"d0e477d9-b58a-4766-b341-ef28fa5d69ab\") " pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:23:26.763610 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.763593    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:23:26.774284 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.774262    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:23:26.784481 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.784431    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7l7c\" (UniqueName: \"kubernetes.io/projected/d0e477d9-b58a-4766-b341-ef28fa5d69ab-kube-api-access-z7l7c\") pod \"network-check-target-9xqkb\" (UID: \"d0e477d9-b58a-4766-b341-ef28fa5d69ab\") " pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:23:26.847783 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.847704    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x4v4p\""
May 25 11:23:26.855174 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.855154    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:23:26.976359 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:26.976328    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9xqkb"]
May 25 11:23:26.979645 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:23:26.979613    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e477d9_b58a_4766_b341_ef28fa5d69ab.slice/crio-d316f35d3ae7b164368a4403c03c4aff42dce00b4aedff44f40ea01df5b3e1d4 WatchSource:0}: Error finding container d316f35d3ae7b164368a4403c03c4aff42dce00b4aedff44f40ea01df5b3e1d4: Status 404 returned error can't find the container with id d316f35d3ae7b164368a4403c03c4aff42dce00b4aedff44f40ea01df5b3e1d4
May 25 11:23:27.175298 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:27.175208    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9xqkb" event={"ID":"d0e477d9-b58a-4766-b341-ef28fa5d69ab","Type":"ContainerStarted","Data":"d316f35d3ae7b164368a4403c03c4aff42dce00b4aedff44f40ea01df5b3e1d4"}
May 25 11:23:31.185788 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:31.185747    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9xqkb" event={"ID":"d0e477d9-b58a-4766-b341-ef28fa5d69ab","Type":"ContainerStarted","Data":"e5229b5f67617b66fec2b513b2b3244f275852bd16b661e7d6c01b288788d12d"}
May 25 11:23:31.186151 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:31.185888    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:23:31.204203 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:31.204155    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9xqkb" podStartSLOduration=66.930196443 podStartE2EDuration="1m10.204139528s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:23:26.98153895 +0000 UTC m=+66.630823926" lastFinishedPulling="2026-05-25 11:23:30.255482021 +0000 UTC m=+69.904767011" observedRunningTime="2026-05-25 11:23:31.203380311 +0000 UTC m=+70.852665304" watchObservedRunningTime="2026-05-25 11:23:31.204139528 +0000 UTC m=+70.853424517"
May 25 11:23:58.600164 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:58.600118    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:23:58.600164 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:58.600164    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:23:58.600789 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:23:58.600191    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:23:58.600789 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:58.600298    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:23:58.600789 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:58.600303    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:23:58.600789 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:58.600298    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:23:58.600789 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:58.600372    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert podName:f6151b3e-2de0-49db-a40a-3a4d0e75c3e1 nodeName:}" failed. No retries permitted until 2026-05-25 11:25:02.600356543 +0000 UTC m=+162.249641521 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert") pod "ingress-canary-h5vl5" (UID: "f6151b3e-2de0-49db-a40a-3a4d0e75c3e1") : secret "canary-serving-cert" not found
May 25 11:23:58.600789 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:58.600410    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls podName:f0720f3e-ce4d-4fc2-a5de-5b624520ae3c nodeName:}" failed. No retries permitted until 2026-05-25 11:25:02.600378816 +0000 UTC m=+162.249663802 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls") pod "dns-default-rlr4p" (UID: "f0720f3e-ce4d-4fc2-a5de-5b624520ae3c") : secret "dns-default-metrics-tls" not found
May 25 11:23:58.600789 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:58.600315    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-756c5b7b96-tnzsx: secret "image-registry-tls" not found
May 25 11:23:58.600789 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:23:58.600511    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls podName:7d43b17c-c5b2-419a-b6a2-06e83911b9ca nodeName:}" failed. No retries permitted until 2026-05-25 11:25:02.600497052 +0000 UTC m=+162.249782028 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls") pod "image-registry-756c5b7b96-tnzsx" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca") : secret "image-registry-tls" not found
May 25 11:24:02.190804 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:24:02.190769    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9xqkb"
May 25 11:24:30.512865 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:24:30.512840    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wkkdr_6375e270-ce2d-4549-a415-343be34c4dfc/dns-node-resolver/0.log"
May 25 11:24:30.737077 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:24:30.737040    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:24:30.737250 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:24:30.737202    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 11:24:30.737291 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:24:30.737280    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs podName:489b5117-49e0-45e5-b99a-3ad57febc838 nodeName:}" failed. No retries permitted until 2026-05-25 11:26:32.737263549 +0000 UTC m=+252.386548526 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs") pod "network-metrics-daemon-tpl95" (UID: "489b5117-49e0-45e5-b99a-3ad57febc838") : secret "metrics-daemon-secret" not found
May 25 11:24:31.313008 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:24:31.312985    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-57jwf_ac7c52f6-4dec-48a3-a106-0fb515461899/node-ca/0.log"
May 25 11:24:57.696518 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:24:57.696449    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx" podUID="7d43b17c-c5b2-419a-b6a2-06e83911b9ca"
May 25 11:24:57.713710 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:24:57.713682    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rlr4p" podUID="f0720f3e-ce4d-4fc2-a5de-5b624520ae3c"
May 25 11:24:57.719858 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:24:57.719820    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-h5vl5" podUID="f6151b3e-2de0-49db-a40a-3a4d0e75c3e1"
May 25 11:24:58.388441 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:24:58.388406    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:24:58.388441 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:24:58.388435    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rlr4p"
May 25 11:24:58.388441 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:24:58.388446    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:24:58.951247 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:24:58.951204    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-tpl95" podUID="489b5117-49e0-45e5-b99a-3ad57febc838"
May 25 11:25:02.681437 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.681390    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:25:02.681437 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.681431    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:25:02.681879 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.681452    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:25:02.683906 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.683874    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f0720f3e-ce4d-4fc2-a5de-5b624520ae3c-metrics-tls\") pod \"dns-default-rlr4p\" (UID: \"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c\") " pod="openshift-dns/dns-default-rlr4p"
May 25 11:25:02.683906 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.683897    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6151b3e-2de0-49db-a40a-3a4d0e75c3e1-cert\") pod \"ingress-canary-h5vl5\" (UID: \"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1\") " pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:25:02.684051 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.683963    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"image-registry-756c5b7b96-tnzsx\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") " pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:25:02.893265 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.893233    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mbq6k\""
May 25 11:25:02.893265 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.893234    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n7v95\""
May 25 11:25:02.893528 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.893274    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w59mk\""
May 25 11:25:02.899681 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.899657    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:25:02.899681 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.899674    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h5vl5"
May 25 11:25:02.899818 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:02.899667    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rlr4p"
May 25 11:25:03.054689 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.054656    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h5vl5"]
May 25 11:25:03.057499 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:25:03.057472    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6151b3e_2de0_49db_a40a_3a4d0e75c3e1.slice/crio-663f643ca51bbbd5ec1a67c63544902684136621188d521478ffee8fb6b4ad9f WatchSource:0}: Error finding container 663f643ca51bbbd5ec1a67c63544902684136621188d521478ffee8fb6b4ad9f: Status 404 returned error can't find the container with id 663f643ca51bbbd5ec1a67c63544902684136621188d521478ffee8fb6b4ad9f
May 25 11:25:03.278389 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.278353    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rlr4p"]
May 25 11:25:03.279282 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.279236    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-756c5b7b96-tnzsx"]
May 25 11:25:03.282701 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:25:03.282667    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d43b17c_c5b2_419a_b6a2_06e83911b9ca.slice/crio-c666ccbf64e1e57e46b451e3e646a8ac62b228b4835e9987610848b743ecea92 WatchSource:0}: Error finding container c666ccbf64e1e57e46b451e3e646a8ac62b228b4835e9987610848b743ecea92: Status 404 returned error can't find the container with id c666ccbf64e1e57e46b451e3e646a8ac62b228b4835e9987610848b743ecea92
May 25 11:25:03.283271 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:25:03.283245    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0720f3e_ce4d_4fc2_a5de_5b624520ae3c.slice/crio-bd807ad66f62761bc05474f86928f66a2738f91fc880c8eea23886884f5263ee WatchSource:0}: Error finding container bd807ad66f62761bc05474f86928f66a2738f91fc880c8eea23886884f5263ee: Status 404 returned error can't find the container with id bd807ad66f62761bc05474f86928f66a2738f91fc880c8eea23886884f5263ee
May 25 11:25:03.402414 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.402368    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx" event={"ID":"7d43b17c-c5b2-419a-b6a2-06e83911b9ca","Type":"ContainerStarted","Data":"121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d"}
May 25 11:25:03.402414 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.402415    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx" event={"ID":"7d43b17c-c5b2-419a-b6a2-06e83911b9ca","Type":"ContainerStarted","Data":"c666ccbf64e1e57e46b451e3e646a8ac62b228b4835e9987610848b743ecea92"}
May 25 11:25:03.402684 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.402503    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:25:03.403506 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.403482    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h5vl5" event={"ID":"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1","Type":"ContainerStarted","Data":"663f643ca51bbbd5ec1a67c63544902684136621188d521478ffee8fb6b4ad9f"}
May 25 11:25:03.404534 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.404512    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlr4p" event={"ID":"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c","Type":"ContainerStarted","Data":"bd807ad66f62761bc05474f86928f66a2738f91fc880c8eea23886884f5263ee"}
May 25 11:25:03.424285 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.424246    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xl9q7"]
May 25 11:25:03.426243 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.426217    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.430992 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.430966    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 11:25:03.431285 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.431266    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s56fk\""
May 25 11:25:03.431383 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.431271    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 11:25:03.431491 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.431447    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 11:25:03.432052 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.432033    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 11:25:03.444474 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.444399    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx" podStartSLOduration=162.444381754 podStartE2EDuration="2m42.444381754s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:25:03.441044136 +0000 UTC m=+163.090329130" watchObservedRunningTime="2026-05-25 11:25:03.444381754 +0000 UTC m=+163.093666750"
May 25 11:25:03.445000 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.444969    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xl9q7"]
May 25 11:25:03.485890 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.485855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-crio-socket\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.486125 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.485914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-data-volume\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.486125 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.486068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.486240 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.486152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlkn\" (UniqueName: \"kubernetes.io/projected/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-kube-api-access-6xlkn\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.486240 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.486180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.587623 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.587537    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.587623 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.587616    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlkn\" (UniqueName: \"kubernetes.io/projected/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-kube-api-access-6xlkn\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.587839 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.587647    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.587839 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.587683    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-crio-socket\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.587839 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.587727    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-data-volume\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.588060 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.588041    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-data-volume\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.588124 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.588063    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-crio-socket\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.588666 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.588510    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.590391 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.590364    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.596906 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.596876    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlkn\" (UniqueName: \"kubernetes.io/projected/6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf-kube-api-access-6xlkn\") pod \"insights-runtime-extractor-xl9q7\" (UID: \"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf\") " pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.735374 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.735335    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xl9q7"
May 25 11:25:03.897704 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:03.897646    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xl9q7"]
May 25 11:25:03.901979 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:25:03.901943    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a306024_ef9e_4f8d_bdd5_4a77fbf6eeaf.slice/crio-3b463bfa431bc23af18e49913beaaf99638a35366b8f30fba46789a778779d8c WatchSource:0}: Error finding container 3b463bfa431bc23af18e49913beaaf99638a35366b8f30fba46789a778779d8c: Status 404 returned error can't find the container with id 3b463bfa431bc23af18e49913beaaf99638a35366b8f30fba46789a778779d8c
May 25 11:25:04.410058 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:04.410020    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xl9q7" event={"ID":"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf","Type":"ContainerStarted","Data":"3413b99f302d42013af018ed101794f79b4c0da5203fe82040a97dbe704352c0"}
May 25 11:25:04.410254 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:04.410066    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xl9q7" event={"ID":"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf","Type":"ContainerStarted","Data":"3b463bfa431bc23af18e49913beaaf99638a35366b8f30fba46789a778779d8c"}
May 25 11:25:05.105551 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:05.105490    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj" podUID="70bb9cc9-7bb3-443b-823d-432d43967d4b" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.6:8000/readyz\": dial tcp 10.132.0.6:8000: connect: connection refused"
May 25 11:25:05.414655 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:05.414626    2578 generic.go:358] "Generic (PLEG): container finished" podID="ec1becf5-75b4-480e-8f33-f131f7186bf5" containerID="54ab2ec7779c490aa960dde1ebd61bf6349bc7db71ccf0996994f214800388f6" exitCode=255
May 25 11:25:05.414789 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:05.414684    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m" event={"ID":"ec1becf5-75b4-480e-8f33-f131f7186bf5","Type":"ContainerDied","Data":"54ab2ec7779c490aa960dde1ebd61bf6349bc7db71ccf0996994f214800388f6"}
May 25 11:25:05.414946 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:05.414934    2578 scope.go:117] "RemoveContainer" containerID="54ab2ec7779c490aa960dde1ebd61bf6349bc7db71ccf0996994f214800388f6"
May 25 11:25:05.417108 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:05.417049    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlr4p" event={"ID":"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c","Type":"ContainerStarted","Data":"5e28a13daa046d091524fc1e1c2e95cf5e9f105c717b8958d27a5d69c4bc150d"}
May 25 11:25:05.418356 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:05.418335    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h5vl5" event={"ID":"f6151b3e-2de0-49db-a40a-3a4d0e75c3e1","Type":"ContainerStarted","Data":"1332dbd9f9b30f159c7264ff0d7ce0b86cd13b0b742666304f67cdf96c0d07ca"}
May 25 11:25:05.420272 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:05.420248    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xl9q7" event={"ID":"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf","Type":"ContainerStarted","Data":"6ae4ef34fba10074f2a9b6825bef098e868064c28c1f3de08bccd284184b4d24"}
May 25 11:25:05.423678 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:05.423651    2578 generic.go:358] "Generic (PLEG): container finished" podID="70bb9cc9-7bb3-443b-823d-432d43967d4b" containerID="8a4c08fa915b3d7fe326a30b3998b6dbff3f0b87fb87beecdebe2c657ddf83fa" exitCode=1
May 25 11:25:05.423737 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:05.423707    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj" event={"ID":"70bb9cc9-7bb3-443b-823d-432d43967d4b","Type":"ContainerDied","Data":"8a4c08fa915b3d7fe326a30b3998b6dbff3f0b87fb87beecdebe2c657ddf83fa"}
May 25 11:25:05.424077 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:05.424058    2578 scope.go:117] "RemoveContainer" containerID="8a4c08fa915b3d7fe326a30b3998b6dbff3f0b87fb87beecdebe2c657ddf83fa"
May 25 11:25:05.468255 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:05.468212    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h5vl5" podStartSLOduration=129.276508351 podStartE2EDuration="2m11.468197738s" podCreationTimestamp="2026-05-25 11:22:54 +0000 UTC" firstStartedPulling="2026-05-25 11:25:03.059203666 +0000 UTC m=+162.708488639" lastFinishedPulling="2026-05-25 11:25:05.250893035 +0000 UTC m=+164.900178026" observedRunningTime="2026-05-25 11:25:05.467770142 +0000 UTC m=+165.117055138" watchObservedRunningTime="2026-05-25 11:25:05.468197738 +0000 UTC m=+165.117482733"
May 25 11:25:06.428501 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:06.428443    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj" event={"ID":"70bb9cc9-7bb3-443b-823d-432d43967d4b","Type":"ContainerStarted","Data":"8d1100805b4afe139374148e397d9b151e34b8767c139ed8030b2fa8d5c356e4"}
May 25 11:25:06.428930 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:06.428839    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:25:06.429534 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:06.429513    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598b6b76dc-hbkwj"
May 25 11:25:06.430012 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:06.429994    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bcd655556-l4w8m" event={"ID":"ec1becf5-75b4-480e-8f33-f131f7186bf5","Type":"ContainerStarted","Data":"fb56e690cf7b052ed9b64e20c0e7b4f098f234905d6be31e01fd811b4af33985"}
May 25 11:25:06.431504 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:06.431481    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlr4p" event={"ID":"f0720f3e-ce4d-4fc2-a5de-5b624520ae3c","Type":"ContainerStarted","Data":"98ab58256c1e2b5a84237764797de23acf626c051c752e7e5a99dc45a233e416"}
May 25 11:25:06.431609 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:06.431563    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rlr4p"
May 25 11:25:06.498444 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:06.498391    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-rlr4p" podStartSLOduration=130.532613949 podStartE2EDuration="2m12.498371019s" podCreationTimestamp="2026-05-25 11:22:54 +0000 UTC" firstStartedPulling="2026-05-25 11:25:03.285132006 +0000 UTC m=+162.934416980" lastFinishedPulling="2026-05-25 11:25:05.250889077 +0000 UTC m=+164.900174050" observedRunningTime="2026-05-25 11:25:06.496901406 +0000 UTC m=+166.146186402" watchObservedRunningTime="2026-05-25 11:25:06.498371019 +0000 UTC m=+166.147656015"
May 25 11:25:07.435432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:07.435396    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xl9q7" event={"ID":"6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf","Type":"ContainerStarted","Data":"cf78427cc0a8bb54732ad447853676338ca9d839bae8d0e6001434a8ee61b19d"}
May 25 11:25:07.458833 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:07.458786    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xl9q7" podStartSLOduration=1.996249318 podStartE2EDuration="4.458770301s" podCreationTimestamp="2026-05-25 11:25:03 +0000 UTC" firstStartedPulling="2026-05-25 11:25:03.979877178 +0000 UTC m=+163.629162159" lastFinishedPulling="2026-05-25 11:25:06.442398167 +0000 UTC m=+166.091683142" observedRunningTime="2026-05-25 11:25:07.456883721 +0000 UTC m=+167.106168730" watchObservedRunningTime="2026-05-25 11:25:07.458770301 +0000 UTC m=+167.108055296"
May 25 11:25:09.700941 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.700857    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bh2c7"]
May 25 11:25:09.703498 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.703474    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.705972 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.705948    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 11:25:09.706066 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.705991    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 11:25:09.707238 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.707190    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 11:25:09.707603 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.707244    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 11:25:09.707603 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.707253    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 11:25:09.707603 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.707265    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 11:25:09.707603 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.707333    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s6tg8\""
May 25 11:25:09.842530 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.842500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.842530 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.842538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-sys\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.842750 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.842562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-wtmp\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.842750 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.842605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-metrics-client-ca\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.842750 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.842651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9fk\" (UniqueName: \"kubernetes.io/projected/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-kube-api-access-bz9fk\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.842750 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.842691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-accelerators-collector-config\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.842944 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.842749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-root\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.842944 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.842802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-textfile\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.842944 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.842829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-tls\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.943849 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.943816    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-wtmp\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944038 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.943879    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-metrics-client-ca\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944038 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.943897    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9fk\" (UniqueName: \"kubernetes.io/projected/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-kube-api-access-bz9fk\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944038 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.943921    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-accelerators-collector-config\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944038 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.943993    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-wtmp\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944233 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.944145    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-root\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944233 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.944198    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-root\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944233 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.944212    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-textfile\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944369 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.944242    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-tls\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944369 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.944287    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944369 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.944316    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-sys\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944544 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.944386    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-sys\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944594 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.944569    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-textfile\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944668 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.944631    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-accelerators-collector-config\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.944668 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.944639    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-metrics-client-ca\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.946947 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.946918    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-tls\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.947052 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.946952    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:09.952232 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:09.952182    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9fk\" (UniqueName: \"kubernetes.io/projected/dd04e54a-1fc4-4a20-a894-76b3935e0bdb-kube-api-access-bz9fk\") pod \"node-exporter-bh2c7\" (UID: \"dd04e54a-1fc4-4a20-a894-76b3935e0bdb\") " pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:10.012166 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:10.012127    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bh2c7"
May 25 11:25:10.021678 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:25:10.021647    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd04e54a_1fc4_4a20_a894_76b3935e0bdb.slice/crio-2096f17fb690359b04f64ced7990c46bf7b9a5acd8ab81cc5dab151e84351f8a WatchSource:0}: Error finding container 2096f17fb690359b04f64ced7990c46bf7b9a5acd8ab81cc5dab151e84351f8a: Status 404 returned error can't find the container with id 2096f17fb690359b04f64ced7990c46bf7b9a5acd8ab81cc5dab151e84351f8a
May 25 11:25:10.444650 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:10.444616    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bh2c7" event={"ID":"dd04e54a-1fc4-4a20-a894-76b3935e0bdb","Type":"ContainerStarted","Data":"2096f17fb690359b04f64ced7990c46bf7b9a5acd8ab81cc5dab151e84351f8a"}
May 25 11:25:10.932328 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:10.932296    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:25:11.448250 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:11.448217    2578 generic.go:358] "Generic (PLEG): container finished" podID="dd04e54a-1fc4-4a20-a894-76b3935e0bdb" containerID="fb2873a6a8955e0b8bbeb6a0f97d95fb77a4f692c76b7bcb6e80580480cc3ca8" exitCode=0
May 25 11:25:11.448434 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:11.448270    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bh2c7" event={"ID":"dd04e54a-1fc4-4a20-a894-76b3935e0bdb","Type":"ContainerDied","Data":"fb2873a6a8955e0b8bbeb6a0f97d95fb77a4f692c76b7bcb6e80580480cc3ca8"}
May 25 11:25:12.452862 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:12.452824    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bh2c7" event={"ID":"dd04e54a-1fc4-4a20-a894-76b3935e0bdb","Type":"ContainerStarted","Data":"bcd77a284b3c1b9d58f31c4461a72fc783066ae3c3a12ee24f0fcefe249216b1"}
May 25 11:25:12.453240 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:12.452868    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bh2c7" event={"ID":"dd04e54a-1fc4-4a20-a894-76b3935e0bdb","Type":"ContainerStarted","Data":"a82f4c2592fc27a6bb208b0fae59119fe9e3bf6f8ac6ab1a80e3265f77200894"}
May 25 11:25:12.473913 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:12.473865    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bh2c7" podStartSLOduration=2.524311806 podStartE2EDuration="3.473850368s" podCreationTimestamp="2026-05-25 11:25:09 +0000 UTC" firstStartedPulling="2026-05-25 11:25:10.023484021 +0000 UTC m=+169.672768995" lastFinishedPulling="2026-05-25 11:25:10.973022582 +0000 UTC m=+170.622307557" observedRunningTime="2026-05-25 11:25:12.472507303 +0000 UTC m=+172.121792300" watchObservedRunningTime="2026-05-25 11:25:12.473850368 +0000 UTC m=+172.123135362"
May 25 11:25:16.437636 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:16.437603    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rlr4p"
May 25 11:25:22.904046 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:22.904007    2578 patch_prober.go:28] interesting pod/image-registry-756c5b7b96-tnzsx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 11:25:22.904512 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:22.904063    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx" podUID="7d43b17c-c5b2-419a-b6a2-06e83911b9ca" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:25:24.414414 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:24.414386    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:25:25.242067 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:25.242025    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-756c5b7b96-tnzsx"]
May 25 11:25:44.983171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:44.983129    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" podUID="38fd9e26-33a6-4a32-b382-de9a5433b29e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 11:25:50.261001 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.260965    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx" podUID="7d43b17c-c5b2-419a-b6a2-06e83911b9ca" containerName="registry" containerID="cri-o://121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d" gracePeriod=30
May 25 11:25:50.493725 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.493701    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:25:50.535929 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.535862    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-installation-pull-secrets\") pod \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") "
May 25 11:25:50.535929 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.535907    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-image-registry-private-configuration\") pod \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") "
May 25 11:25:50.536095 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.535930    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-trusted-ca\") pod \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") "
May 25 11:25:50.536095 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.535969    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-ca-trust-extracted\") pod \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") "
May 25 11:25:50.536095 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.535993    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vlrb\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-kube-api-access-4vlrb\") pod \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") "
May 25 11:25:50.536095 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.536022    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-bound-sa-token\") pod \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") "
May 25 11:25:50.536290 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.536109    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") pod \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") "
May 25 11:25:50.536290 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.536184    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-certificates\") pod \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\" (UID: \"7d43b17c-c5b2-419a-b6a2-06e83911b9ca\") "
May 25 11:25:50.536743 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.536686    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7d43b17c-c5b2-419a-b6a2-06e83911b9ca" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:25:50.536865 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.536738    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7d43b17c-c5b2-419a-b6a2-06e83911b9ca" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:25:50.538806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.538770    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7d43b17c-c5b2-419a-b6a2-06e83911b9ca" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:50.538938 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.538908    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-kube-api-access-4vlrb" (OuterVolumeSpecName: "kube-api-access-4vlrb") pod "7d43b17c-c5b2-419a-b6a2-06e83911b9ca" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca"). InnerVolumeSpecName "kube-api-access-4vlrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:50.539017 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.538947    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7d43b17c-c5b2-419a-b6a2-06e83911b9ca" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:25:50.539017 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.538964    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7d43b17c-c5b2-419a-b6a2-06e83911b9ca" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:50.539117 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.539023    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7d43b17c-c5b2-419a-b6a2-06e83911b9ca" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:25:50.544341 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.544318    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7d43b17c-c5b2-419a-b6a2-06e83911b9ca" (UID: "7d43b17c-c5b2-419a-b6a2-06e83911b9ca"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:50.557151 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.557126    2578 generic.go:358] "Generic (PLEG): container finished" podID="7d43b17c-c5b2-419a-b6a2-06e83911b9ca" containerID="121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d" exitCode=0
May 25 11:25:50.557248 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.557186    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx"
May 25 11:25:50.557248 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.557196    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx" event={"ID":"7d43b17c-c5b2-419a-b6a2-06e83911b9ca","Type":"ContainerDied","Data":"121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d"}
May 25 11:25:50.557248 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.557218    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-756c5b7b96-tnzsx" event={"ID":"7d43b17c-c5b2-419a-b6a2-06e83911b9ca","Type":"ContainerDied","Data":"c666ccbf64e1e57e46b451e3e646a8ac62b228b4835e9987610848b743ecea92"}
May 25 11:25:50.557248 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.557232    2578 scope.go:117] "RemoveContainer" containerID="121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d"
May 25 11:25:50.565329 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.565310    2578 scope.go:117] "RemoveContainer" containerID="121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d"
May 25 11:25:50.565641 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:25:50.565614    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d\": container with ID starting with 121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d not found: ID does not exist" containerID="121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d"
May 25 11:25:50.565722 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.565648    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d"} err="failed to get container status \"121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d\": rpc error: code = NotFound desc = could not find container \"121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d\": container with ID starting with 121918928e11ea90d065200bb00836110d0190b77b7b58480cfe431fc0b9759d not found: ID does not exist"
May 25 11:25:50.578430 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.578406    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-756c5b7b96-tnzsx"]
May 25 11:25:50.582632 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.582605    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-756c5b7b96-tnzsx"]
May 25 11:25:50.637114 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.637070    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-certificates\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:25:50.637114 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.637111    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-installation-pull-secrets\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:25:50.637313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.637124    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-image-registry-private-configuration\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:25:50.637313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.637133    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-trusted-ca\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:25:50.637313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.637143    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-ca-trust-extracted\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:25:50.637313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.637151    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vlrb\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-kube-api-access-4vlrb\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:25:50.637313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.637161    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-bound-sa-token\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:25:50.637313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.637170    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d43b17c-c5b2-419a-b6a2-06e83911b9ca-registry-tls\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:25:50.934412 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:50.934335    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d43b17c-c5b2-419a-b6a2-06e83911b9ca" path="/var/lib/kubelet/pods/7d43b17c-c5b2-419a-b6a2-06e83911b9ca/volumes"
May 25 11:25:54.983610 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:25:54.983569    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" podUID="38fd9e26-33a6-4a32-b382-de9a5433b29e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 11:26:04.983544 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:04.983503    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" podUID="38fd9e26-33a6-4a32-b382-de9a5433b29e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 11:26:04.983922 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:04.983574    2578 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br"
May 25 11:26:04.984037 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:04.984020    2578 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6e810f1b93e895d832506f8679e3cea5828a27dcea51dc11c271ff54566a44d7"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 25 11:26:04.984074 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:04.984059    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" podUID="38fd9e26-33a6-4a32-b382-de9a5433b29e" containerName="service-proxy" containerID="cri-o://6e810f1b93e895d832506f8679e3cea5828a27dcea51dc11c271ff54566a44d7" gracePeriod=30
May 25 11:26:05.597690 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:05.597658    2578 generic.go:358] "Generic (PLEG): container finished" podID="38fd9e26-33a6-4a32-b382-de9a5433b29e" containerID="6e810f1b93e895d832506f8679e3cea5828a27dcea51dc11c271ff54566a44d7" exitCode=2
May 25 11:26:05.597859 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:05.597728    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" event={"ID":"38fd9e26-33a6-4a32-b382-de9a5433b29e","Type":"ContainerDied","Data":"6e810f1b93e895d832506f8679e3cea5828a27dcea51dc11c271ff54566a44d7"}
May 25 11:26:05.597859 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:05.597766    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77654fcb55-cv7br" event={"ID":"38fd9e26-33a6-4a32-b382-de9a5433b29e","Type":"ContainerStarted","Data":"c70215fd671f0e0d8394215c6e10fa31aaf5930af48e52451b70e8b8daaeafa0"}
May 25 11:26:32.738395 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:32.738336    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:26:32.740720 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:32.740697    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/489b5117-49e0-45e5-b99a-3ad57febc838-metrics-certs\") pod \"network-metrics-daemon-tpl95\" (UID: \"489b5117-49e0-45e5-b99a-3ad57febc838\") " pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:26:32.836634 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:32.836604    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-brg2t\""
May 25 11:26:32.843901 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:32.843876    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tpl95"
May 25 11:26:32.971194 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:32.971165    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tpl95"]
May 25 11:26:32.973665 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:26:32.973641    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489b5117_49e0_45e5_b99a_3ad57febc838.slice/crio-00ab13f225913dc376c71279b726a3b9258db70ee937e946948c8839c7e922a3 WatchSource:0}: Error finding container 00ab13f225913dc376c71279b726a3b9258db70ee937e946948c8839c7e922a3: Status 404 returned error can't find the container with id 00ab13f225913dc376c71279b726a3b9258db70ee937e946948c8839c7e922a3
May 25 11:26:33.668255 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:33.668212    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tpl95" event={"ID":"489b5117-49e0-45e5-b99a-3ad57febc838","Type":"ContainerStarted","Data":"00ab13f225913dc376c71279b726a3b9258db70ee937e946948c8839c7e922a3"}
May 25 11:26:34.672312 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:34.672274    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tpl95" event={"ID":"489b5117-49e0-45e5-b99a-3ad57febc838","Type":"ContainerStarted","Data":"6fa8766bb166c9edcf8b6d633a719820d5390d589371e5905543895eb1402e92"}
May 25 11:26:34.672312 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:34.672314    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tpl95" event={"ID":"489b5117-49e0-45e5-b99a-3ad57febc838","Type":"ContainerStarted","Data":"b011864ab6603c5fd1059a44b99ebfc1f8e1b84dfe8895195e9ec2d12289578d"}
May 25 11:26:34.693838 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:26:34.693792    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tpl95" podStartSLOduration=252.768577038 podStartE2EDuration="4m13.693778481s" podCreationTimestamp="2026-05-25 11:22:21 +0000 UTC" firstStartedPulling="2026-05-25 11:26:32.975337543 +0000 UTC m=+252.624622519" lastFinishedPulling="2026-05-25 11:26:33.900538985 +0000 UTC m=+253.549823962" observedRunningTime="2026-05-25 11:26:34.692142456 +0000 UTC m=+254.341427451" watchObservedRunningTime="2026-05-25 11:26:34.693778481 +0000 UTC m=+254.343063476"
May 25 11:27:20.858008 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:20.857980    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:27:20.861874 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:20.861849    2578 kubelet.go:1628] "Image garbage collection succeeded"
May 25 11:27:20.862184 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:20.862167    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:27:37.674685 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.674642    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"]
May 25 11:27:37.677061 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.674941    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d43b17c-c5b2-419a-b6a2-06e83911b9ca" containerName="registry"
May 25 11:27:37.677061 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.674958    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d43b17c-c5b2-419a-b6a2-06e83911b9ca" containerName="registry"
May 25 11:27:37.677061 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.675033    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="7d43b17c-c5b2-419a-b6a2-06e83911b9ca" containerName="registry"
May 25 11:27:37.677917 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.677896    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:37.680723 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.680697    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:27:37.680827 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.680727    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:27:37.681887 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.681858    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6sjvf\""
May 25 11:27:37.686430 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.686411    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"]
May 25 11:27:37.775369 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.775334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:37.775574 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.775380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:37.775574 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.775443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjc9w\" (UniqueName: \"kubernetes.io/projected/6de277cd-0afb-48c2-8544-6407fe180c3d-kube-api-access-bjc9w\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:37.876119 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.876081    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bjc9w\" (UniqueName: \"kubernetes.io/projected/6de277cd-0afb-48c2-8544-6407fe180c3d-kube-api-access-bjc9w\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:37.876119 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.876124    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:37.876300 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.876153    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:37.876525 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.876505    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:37.876565 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.876531    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:37.886957 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.886935    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjc9w\" (UniqueName: \"kubernetes.io/projected/6de277cd-0afb-48c2-8544-6407fe180c3d-kube-api-access-bjc9w\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:37.986872 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:37.986827    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:38.115233 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:38.115207    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"]
May 25 11:27:38.117842 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:27:38.117818    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de277cd_0afb_48c2_8544_6407fe180c3d.slice/crio-7fef7db3c1a2a7e317e64c837f91310a9cc3ec4a228aeb3eeb0d0cac197ab89f WatchSource:0}: Error finding container 7fef7db3c1a2a7e317e64c837f91310a9cc3ec4a228aeb3eeb0d0cac197ab89f: Status 404 returned error can't find the container with id 7fef7db3c1a2a7e317e64c837f91310a9cc3ec4a228aeb3eeb0d0cac197ab89f
May 25 11:27:38.119566 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:38.119551    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:27:38.835016 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:38.834978    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv" event={"ID":"6de277cd-0afb-48c2-8544-6407fe180c3d","Type":"ContainerStarted","Data":"7fef7db3c1a2a7e317e64c837f91310a9cc3ec4a228aeb3eeb0d0cac197ab89f"}
May 25 11:27:44.851358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:44.851269    2578 generic.go:358] "Generic (PLEG): container finished" podID="6de277cd-0afb-48c2-8544-6407fe180c3d" containerID="afd55f468c275ac5f1c616e55879343e50769ad4ef93a7027d0068de49e2e392" exitCode=0
May 25 11:27:44.851358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:44.851334    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv" event={"ID":"6de277cd-0afb-48c2-8544-6407fe180c3d","Type":"ContainerDied","Data":"afd55f468c275ac5f1c616e55879343e50769ad4ef93a7027d0068de49e2e392"}
May 25 11:27:46.859204 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:46.859174    2578 generic.go:358] "Generic (PLEG): container finished" podID="6de277cd-0afb-48c2-8544-6407fe180c3d" containerID="459299c818a771d8e746e671750546b7708221e7338dd71287814ad0b2902b46" exitCode=0
May 25 11:27:46.859586 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:46.859231    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv" event={"ID":"6de277cd-0afb-48c2-8544-6407fe180c3d","Type":"ContainerDied","Data":"459299c818a771d8e746e671750546b7708221e7338dd71287814ad0b2902b46"}
May 25 11:27:53.878478 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:53.878421    2578 generic.go:358] "Generic (PLEG): container finished" podID="6de277cd-0afb-48c2-8544-6407fe180c3d" containerID="d675c990ad4564e2f7f35b5de7241dfbc7bd80e2ef277d67e07fffc5e91de2c7" exitCode=0
May 25 11:27:53.878844 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:53.878500    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv" event={"ID":"6de277cd-0afb-48c2-8544-6407fe180c3d","Type":"ContainerDied","Data":"d675c990ad4564e2f7f35b5de7241dfbc7bd80e2ef277d67e07fffc5e91de2c7"}
May 25 11:27:54.999809 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:54.999787    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:27:55.003089 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.003071    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjc9w\" (UniqueName: \"kubernetes.io/projected/6de277cd-0afb-48c2-8544-6407fe180c3d-kube-api-access-bjc9w\") pod \"6de277cd-0afb-48c2-8544-6407fe180c3d\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") "
May 25 11:27:55.003147 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.003102    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-util\") pod \"6de277cd-0afb-48c2-8544-6407fe180c3d\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") "
May 25 11:27:55.003147 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.003141    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-bundle\") pod \"6de277cd-0afb-48c2-8544-6407fe180c3d\" (UID: \"6de277cd-0afb-48c2-8544-6407fe180c3d\") "
May 25 11:27:55.008897 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.004075    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-bundle" (OuterVolumeSpecName: "bundle") pod "6de277cd-0afb-48c2-8544-6407fe180c3d" (UID: "6de277cd-0afb-48c2-8544-6407fe180c3d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:27:55.009032 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.008933    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-util" (OuterVolumeSpecName: "util") pod "6de277cd-0afb-48c2-8544-6407fe180c3d" (UID: "6de277cd-0afb-48c2-8544-6407fe180c3d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:27:55.009431 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.009406    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de277cd-0afb-48c2-8544-6407fe180c3d-kube-api-access-bjc9w" (OuterVolumeSpecName: "kube-api-access-bjc9w") pod "6de277cd-0afb-48c2-8544-6407fe180c3d" (UID: "6de277cd-0afb-48c2-8544-6407fe180c3d"). InnerVolumeSpecName "kube-api-access-bjc9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:27:55.103747 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.103704    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjc9w\" (UniqueName: \"kubernetes.io/projected/6de277cd-0afb-48c2-8544-6407fe180c3d-kube-api-access-bjc9w\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:27:55.103747 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.103736    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-util\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:27:55.103747 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.103747    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6de277cd-0afb-48c2-8544-6407fe180c3d-bundle\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:27:55.885328 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.885293    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv" event={"ID":"6de277cd-0afb-48c2-8544-6407fe180c3d","Type":"ContainerDied","Data":"7fef7db3c1a2a7e317e64c837f91310a9cc3ec4a228aeb3eeb0d0cac197ab89f"}
May 25 11:27:55.885328 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.885327    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fef7db3c1a2a7e317e64c837f91310a9cc3ec4a228aeb3eeb0d0cac197ab89f"
May 25 11:27:55.885548 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:27:55.885336    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cdjmzv"
May 25 11:28:04.276385 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.276345    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-vrngn"]
May 25 11:28:04.276806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.276581    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de277cd-0afb-48c2-8544-6407fe180c3d" containerName="util"
May 25 11:28:04.276806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.276591    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de277cd-0afb-48c2-8544-6407fe180c3d" containerName="util"
May 25 11:28:04.276806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.276601    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de277cd-0afb-48c2-8544-6407fe180c3d" containerName="extract"
May 25 11:28:04.276806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.276606    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de277cd-0afb-48c2-8544-6407fe180c3d" containerName="extract"
May 25 11:28:04.276806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.276617    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de277cd-0afb-48c2-8544-6407fe180c3d" containerName="pull"
May 25 11:28:04.276806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.276622    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de277cd-0afb-48c2-8544-6407fe180c3d" containerName="pull"
May 25 11:28:04.276806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.276660    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="6de277cd-0afb-48c2-8544-6407fe180c3d" containerName="extract"
May 25 11:28:04.310707 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.310670    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-vrngn"]
May 25 11:28:04.310861 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.310800    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:04.314129 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.314106    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 25 11:28:04.314336 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.314307    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 25 11:28:04.314407 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.314379    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 25 11:28:04.314496 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.314403    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 25 11:28:04.315371 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.315354    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 25 11:28:04.315419 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.315354    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-lrlk9\""
May 25 11:28:04.365335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.365297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:04.365335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.365331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b4fdce22-c1ab-4af2-b315-0122a84f6639-cabundle0\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:04.365565 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.365352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zchqx\" (UniqueName: \"kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-kube-api-access-zchqx\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:04.465926 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.465878    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:04.466165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.465940    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b4fdce22-c1ab-4af2-b315-0122a84f6639-cabundle0\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:04.466165 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.465965    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zchqx\" (UniqueName: \"kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-kube-api-access-zchqx\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:04.466165 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.466015    2578 secret.go:281] references non-existent secret key: ca.crt
May 25 11:28:04.466165 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.466039    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 11:28:04.466165 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.466049    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-vrngn: references non-existent secret key: ca.crt
May 25 11:28:04.466165 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.466104    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates podName:b4fdce22-c1ab-4af2-b315-0122a84f6639 nodeName:}" failed. No retries permitted until 2026-05-25 11:28:04.966087765 +0000 UTC m=+344.615372739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates") pod "keda-operator-ffbb595cb-vrngn" (UID: "b4fdce22-c1ab-4af2-b315-0122a84f6639") : references non-existent secret key: ca.crt
May 25 11:28:04.466616 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.466597    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/b4fdce22-c1ab-4af2-b315-0122a84f6639-cabundle0\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:04.476192 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.476163    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zchqx\" (UniqueName: \"kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-kube-api-access-zchqx\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:04.510990 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.510952    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"]
May 25 11:28:04.523107 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.523084    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:04.530307 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.530282    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"]
May 25 11:28:04.531610 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.531584    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 25 11:28:04.566533 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.566494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:04.566716 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.566612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1189384b-afde-4d2a-a644-79074f1a83ac-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:04.566716 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.566639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h728\" (UniqueName: \"kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-kube-api-access-5h728\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:04.667380 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.667337    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1189384b-afde-4d2a-a644-79074f1a83ac-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:04.667380 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.667383    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5h728\" (UniqueName: \"kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-kube-api-access-5h728\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:04.667688 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.667421    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:04.667688 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.667573    2578 secret.go:281] references non-existent secret key: tls.crt
May 25 11:28:04.667688 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.667590    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 11:28:04.667688 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.667611    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt: references non-existent secret key: tls.crt
May 25 11:28:04.667688 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.667686    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates podName:1189384b-afde-4d2a-a644-79074f1a83ac nodeName:}" failed. No retries permitted until 2026-05-25 11:28:05.167666697 +0000 UTC m=+344.816951673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates") pod "keda-metrics-apiserver-7c9f485588-m6dtt" (UID: "1189384b-afde-4d2a-a644-79074f1a83ac") : references non-existent secret key: tls.crt
May 25 11:28:04.667927 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.667735    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1189384b-afde-4d2a-a644-79074f1a83ac-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:04.716760 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.716721    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h728\" (UniqueName: \"kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-kube-api-access-5h728\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:04.969699 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:04.969615    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:04.969844 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.969769    2578 secret.go:281] references non-existent secret key: ca.crt
May 25 11:28:04.969844 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.969788    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 11:28:04.969844 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.969798    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-vrngn: references non-existent secret key: ca.crt
May 25 11:28:04.969959 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:04.969859    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates podName:b4fdce22-c1ab-4af2-b315-0122a84f6639 nodeName:}" failed. No retries permitted until 2026-05-25 11:28:05.969841081 +0000 UTC m=+345.619126057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates") pod "keda-operator-ffbb595cb-vrngn" (UID: "b4fdce22-c1ab-4af2-b315-0122a84f6639") : references non-existent secret key: ca.crt
May 25 11:28:05.171773 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:05.171734    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:05.171978 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:05.171901    2578 secret.go:281] references non-existent secret key: tls.crt
May 25 11:28:05.171978 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:05.171927    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 11:28:05.171978 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:05.171951    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt: references non-existent secret key: tls.crt
May 25 11:28:05.172136 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:05.172020    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates podName:1189384b-afde-4d2a-a644-79074f1a83ac nodeName:}" failed. No retries permitted until 2026-05-25 11:28:06.171998899 +0000 UTC m=+345.821283872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates") pod "keda-metrics-apiserver-7c9f485588-m6dtt" (UID: "1189384b-afde-4d2a-a644-79074f1a83ac") : references non-existent secret key: tls.crt
May 25 11:28:05.978182 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:05.978145    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:05.978619 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:05.978263    2578 secret.go:281] references non-existent secret key: ca.crt
May 25 11:28:05.978619 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:05.978274    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 11:28:05.978619 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:05.978282    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-vrngn: references non-existent secret key: ca.crt
May 25 11:28:05.978619 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:05.978347    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates podName:b4fdce22-c1ab-4af2-b315-0122a84f6639 nodeName:}" failed. No retries permitted until 2026-05-25 11:28:07.978332333 +0000 UTC m=+347.627617306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates") pod "keda-operator-ffbb595cb-vrngn" (UID: "b4fdce22-c1ab-4af2-b315-0122a84f6639") : references non-existent secret key: ca.crt
May 25 11:28:06.180102 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:06.180060    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:06.180283 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:06.180170    2578 secret.go:281] references non-existent secret key: tls.crt
May 25 11:28:06.180283 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:06.180181    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 11:28:06.180283 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:06.180198    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt: references non-existent secret key: tls.crt
May 25 11:28:06.180283 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:28:06.180244    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates podName:1189384b-afde-4d2a-a644-79074f1a83ac nodeName:}" failed. No retries permitted until 2026-05-25 11:28:08.180230715 +0000 UTC m=+347.829515692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates") pod "keda-metrics-apiserver-7c9f485588-m6dtt" (UID: "1189384b-afde-4d2a-a644-79074f1a83ac") : references non-existent secret key: tls.crt
May 25 11:28:07.995230 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:07.995175    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:07.997857 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:07.997833    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4fdce22-c1ab-4af2-b315-0122a84f6639-certificates\") pod \"keda-operator-ffbb595cb-vrngn\" (UID: \"b4fdce22-c1ab-4af2-b315-0122a84f6639\") " pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:08.197306 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:08.197221    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:08.200081 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:08.200055    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1189384b-afde-4d2a-a644-79074f1a83ac-certificates\") pod \"keda-metrics-apiserver-7c9f485588-m6dtt\" (UID: \"1189384b-afde-4d2a-a644-79074f1a83ac\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:08.220720 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:08.220684    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:08.351586 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:08.351561    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-vrngn"]
May 25 11:28:08.353856 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:28:08.353830    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fdce22_c1ab_4af2_b315_0122a84f6639.slice/crio-dbed2cb5643446ebcf5f7e07273c8be07295ec54ddef688b1f0e47a7ed410064 WatchSource:0}: Error finding container dbed2cb5643446ebcf5f7e07273c8be07295ec54ddef688b1f0e47a7ed410064: Status 404 returned error can't find the container with id dbed2cb5643446ebcf5f7e07273c8be07295ec54ddef688b1f0e47a7ed410064
May 25 11:28:08.434435 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:08.434396    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:08.557041 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:08.556939    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"]
May 25 11:28:08.559502 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:28:08.559452    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1189384b_afde_4d2a_a644_79074f1a83ac.slice/crio-f61ced95a7498410dbdbb81830de860b43ecf1905908b4bf380bb87fb8faa153 WatchSource:0}: Error finding container f61ced95a7498410dbdbb81830de860b43ecf1905908b4bf380bb87fb8faa153: Status 404 returned error can't find the container with id f61ced95a7498410dbdbb81830de860b43ecf1905908b4bf380bb87fb8faa153
May 25 11:28:08.919407 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:08.919308    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt" event={"ID":"1189384b-afde-4d2a-a644-79074f1a83ac","Type":"ContainerStarted","Data":"f61ced95a7498410dbdbb81830de860b43ecf1905908b4bf380bb87fb8faa153"}
May 25 11:28:08.920248 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:08.920219    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-vrngn" event={"ID":"b4fdce22-c1ab-4af2-b315-0122a84f6639","Type":"ContainerStarted","Data":"dbed2cb5643446ebcf5f7e07273c8be07295ec54ddef688b1f0e47a7ed410064"}
May 25 11:28:12.933745 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:12.933710    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt" event={"ID":"1189384b-afde-4d2a-a644-79074f1a83ac","Type":"ContainerStarted","Data":"0fb0198d2d627d97b75bf91b50fe731ae42d2c3eb42dac041b1349467439ef9c"}
May 25 11:28:12.934148 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:12.933758    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:12.934148 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:12.933991    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-vrngn" event={"ID":"b4fdce22-c1ab-4af2-b315-0122a84f6639","Type":"ContainerStarted","Data":"6dbd15c6a5981f47852d50b4ddee08a04452e54cd9921d77327c9481a3b162c9"}
May 25 11:28:12.934148 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:12.934078    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:12.952082 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:12.952036    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt" podStartSLOduration=5.436356511 podStartE2EDuration="8.952020766s" podCreationTimestamp="2026-05-25 11:28:04 +0000 UTC" firstStartedPulling="2026-05-25 11:28:08.560741015 +0000 UTC m=+348.210025991" lastFinishedPulling="2026-05-25 11:28:12.07640527 +0000 UTC m=+351.725690246" observedRunningTime="2026-05-25 11:28:12.950665199 +0000 UTC m=+352.599950195" watchObservedRunningTime="2026-05-25 11:28:12.952020766 +0000 UTC m=+352.601305784"
May 25 11:28:12.972008 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:12.971944    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-vrngn" podStartSLOduration=5.247420323 podStartE2EDuration="8.971926274s" podCreationTimestamp="2026-05-25 11:28:04 +0000 UTC" firstStartedPulling="2026-05-25 11:28:08.355101435 +0000 UTC m=+348.004386409" lastFinishedPulling="2026-05-25 11:28:12.079607386 +0000 UTC m=+351.728892360" observedRunningTime="2026-05-25 11:28:12.970440009 +0000 UTC m=+352.619725003" watchObservedRunningTime="2026-05-25 11:28:12.971926274 +0000 UTC m=+352.621211272"
May 25 11:28:23.941000 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:23.940964    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-m6dtt"
May 25 11:28:33.938634 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:33.938602    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-vrngn"
May 25 11:28:58.623347 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.623311    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"]
May 25 11:28:58.628726 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.628703    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:28:58.631751 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.631729    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:28:58.631887 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.631758    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6sjvf\""
May 25 11:28:58.632713 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.632700    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:28:58.635822 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.635576    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"]
May 25 11:28:58.757841 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.757791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qrn9\" (UniqueName: \"kubernetes.io/projected/e50e5d30-1989-42f6-b325-01b314b60c7b-kube-api-access-4qrn9\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:28:58.757841 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.757844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:28:58.758069 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.757906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:28:58.859143 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.859104    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4qrn9\" (UniqueName: \"kubernetes.io/projected/e50e5d30-1989-42f6-b325-01b314b60c7b-kube-api-access-4qrn9\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:28:58.859322 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.859148    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:28:58.859322 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.859176    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:28:58.859548 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.859529    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-util\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:28:58.859620 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.859573    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-bundle\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:28:58.869284 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.869265    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qrn9\" (UniqueName: \"kubernetes.io/projected/e50e5d30-1989-42f6-b325-01b314b60c7b-kube-api-access-4qrn9\") pod \"2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") " pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:28:58.938220 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:58.938146    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:28:59.066170 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:28:59.066107    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"]
May 25 11:28:59.068757 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:28:59.068731    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50e5d30_1989_42f6_b325_01b314b60c7b.slice/crio-3e1f06c7faabb868af9df29df009ae19774821e50aca6954afed13dce25d656c WatchSource:0}: Error finding container 3e1f06c7faabb868af9df29df009ae19774821e50aca6954afed13dce25d656c: Status 404 returned error can't find the container with id 3e1f06c7faabb868af9df29df009ae19774821e50aca6954afed13dce25d656c
May 25 11:29:00.055939 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:00.055900    2578 generic.go:358] "Generic (PLEG): container finished" podID="e50e5d30-1989-42f6-b325-01b314b60c7b" containerID="c95599d088e5b00679a268fdc0e92963580f3d4418d87db147dbacb1ec8448d1" exitCode=0
May 25 11:29:00.056304 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:00.055987    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw" event={"ID":"e50e5d30-1989-42f6-b325-01b314b60c7b","Type":"ContainerDied","Data":"c95599d088e5b00679a268fdc0e92963580f3d4418d87db147dbacb1ec8448d1"}
May 25 11:29:00.056304 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:00.056022    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw" event={"ID":"e50e5d30-1989-42f6-b325-01b314b60c7b","Type":"ContainerStarted","Data":"3e1f06c7faabb868af9df29df009ae19774821e50aca6954afed13dce25d656c"}
May 25 11:29:01.060065 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:01.059974    2578 generic.go:358] "Generic (PLEG): container finished" podID="e50e5d30-1989-42f6-b325-01b314b60c7b" containerID="59d1d1565a5b97ae229328a72b5b9e0d1b277b40bff6ec9614ef93eccef7214b" exitCode=0
May 25 11:29:01.060421 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:01.060067    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw" event={"ID":"e50e5d30-1989-42f6-b325-01b314b60c7b","Type":"ContainerDied","Data":"59d1d1565a5b97ae229328a72b5b9e0d1b277b40bff6ec9614ef93eccef7214b"}
May 25 11:29:02.064440 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:02.064407    2578 generic.go:358] "Generic (PLEG): container finished" podID="e50e5d30-1989-42f6-b325-01b314b60c7b" containerID="87db3908af6b72a85aa9b2617ddd4cc249070b819b449084fec9f1778cbc642c" exitCode=0
May 25 11:29:02.064816 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:02.064484    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw" event={"ID":"e50e5d30-1989-42f6-b325-01b314b60c7b","Type":"ContainerDied","Data":"87db3908af6b72a85aa9b2617ddd4cc249070b819b449084fec9f1778cbc642c"}
May 25 11:29:03.194680 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:03.194652    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:29:03.294604 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:03.294561    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-bundle\") pod \"e50e5d30-1989-42f6-b325-01b314b60c7b\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") "
May 25 11:29:03.294604 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:03.294602    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-util\") pod \"e50e5d30-1989-42f6-b325-01b314b60c7b\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") "
May 25 11:29:03.294818 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:03.294623    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qrn9\" (UniqueName: \"kubernetes.io/projected/e50e5d30-1989-42f6-b325-01b314b60c7b-kube-api-access-4qrn9\") pod \"e50e5d30-1989-42f6-b325-01b314b60c7b\" (UID: \"e50e5d30-1989-42f6-b325-01b314b60c7b\") "
May 25 11:29:03.295249 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:03.295214    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-bundle" (OuterVolumeSpecName: "bundle") pod "e50e5d30-1989-42f6-b325-01b314b60c7b" (UID: "e50e5d30-1989-42f6-b325-01b314b60c7b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:29:03.296816 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:03.296789    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50e5d30-1989-42f6-b325-01b314b60c7b-kube-api-access-4qrn9" (OuterVolumeSpecName: "kube-api-access-4qrn9") pod "e50e5d30-1989-42f6-b325-01b314b60c7b" (UID: "e50e5d30-1989-42f6-b325-01b314b60c7b"). InnerVolumeSpecName "kube-api-access-4qrn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:29:03.300301 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:03.300279    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-util" (OuterVolumeSpecName: "util") pod "e50e5d30-1989-42f6-b325-01b314b60c7b" (UID: "e50e5d30-1989-42f6-b325-01b314b60c7b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:29:03.395322 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:03.395236    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-bundle\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:29:03.395322 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:03.395262    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e50e5d30-1989-42f6-b325-01b314b60c7b-util\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:29:03.395322 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:03.395272    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qrn9\" (UniqueName: \"kubernetes.io/projected/e50e5d30-1989-42f6-b325-01b314b60c7b-kube-api-access-4qrn9\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:29:04.071689 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:04.071656    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw" event={"ID":"e50e5d30-1989-42f6-b325-01b314b60c7b","Type":"ContainerDied","Data":"3e1f06c7faabb868af9df29df009ae19774821e50aca6954afed13dce25d656c"}
May 25 11:29:04.071689 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:04.071687    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e1f06c7faabb868af9df29df009ae19774821e50aca6954afed13dce25d656c"
May 25 11:29:04.071890 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:04.071702    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/2bb52b92bd31ddf2ebbc335370ac517be50e61a93c2fe375393413c19ds4srw"
May 25 11:29:11.667297 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.667257    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d"]
May 25 11:29:11.667877 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.667597    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50e5d30-1989-42f6-b325-01b314b60c7b" containerName="util"
May 25 11:29:11.667877 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.667616    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50e5d30-1989-42f6-b325-01b314b60c7b" containerName="util"
May 25 11:29:11.667877 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.667634    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50e5d30-1989-42f6-b325-01b314b60c7b" containerName="pull"
May 25 11:29:11.667877 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.667643    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50e5d30-1989-42f6-b325-01b314b60c7b" containerName="pull"
May 25 11:29:11.667877 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.667666    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50e5d30-1989-42f6-b325-01b314b60c7b" containerName="extract"
May 25 11:29:11.667877 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.667674    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50e5d30-1989-42f6-b325-01b314b60c7b" containerName="extract"
May 25 11:29:11.667877 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.667733    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="e50e5d30-1989-42f6-b325-01b314b60c7b" containerName="extract"
May 25 11:29:11.670663 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.670642    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d"
May 25 11:29:11.678332 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.678312    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
May 25 11:29:11.678474 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.678312    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-pcsxh\""
May 25 11:29:11.678605 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.678592    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
May 25 11:29:11.704270 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.704230    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d"]
May 25 11:29:11.755755 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.755714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96spc\" (UniqueName: \"kubernetes.io/projected/7ea541a2-254d-48a0-b060-cb26f5180b6f-kube-api-access-96spc\") pod \"cert-manager-operator-controller-manager-54b9655956-ftq6d\" (UID: \"7ea541a2-254d-48a0-b060-cb26f5180b6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d"
May 25 11:29:11.755930 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.755773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ea541a2-254d-48a0-b060-cb26f5180b6f-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-ftq6d\" (UID: \"7ea541a2-254d-48a0-b060-cb26f5180b6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d"
May 25 11:29:11.856206 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.856152    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-96spc\" (UniqueName: \"kubernetes.io/projected/7ea541a2-254d-48a0-b060-cb26f5180b6f-kube-api-access-96spc\") pod \"cert-manager-operator-controller-manager-54b9655956-ftq6d\" (UID: \"7ea541a2-254d-48a0-b060-cb26f5180b6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d"
May 25 11:29:11.856351 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.856221    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ea541a2-254d-48a0-b060-cb26f5180b6f-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-ftq6d\" (UID: \"7ea541a2-254d-48a0-b060-cb26f5180b6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d"
May 25 11:29:11.856573 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.856558    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ea541a2-254d-48a0-b060-cb26f5180b6f-tmp\") pod \"cert-manager-operator-controller-manager-54b9655956-ftq6d\" (UID: \"7ea541a2-254d-48a0-b060-cb26f5180b6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d"
May 25 11:29:11.876851 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.876814    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96spc\" (UniqueName: \"kubernetes.io/projected/7ea541a2-254d-48a0-b060-cb26f5180b6f-kube-api-access-96spc\") pod \"cert-manager-operator-controller-manager-54b9655956-ftq6d\" (UID: \"7ea541a2-254d-48a0-b060-cb26f5180b6f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d"
May 25 11:29:11.979337 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:11.979308    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d"
May 25 11:29:12.128569 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:12.128543    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d"]
May 25 11:29:12.131007 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:29:12.130977    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea541a2_254d_48a0_b060_cb26f5180b6f.slice/crio-abddd4bd64b1a7fdf9591474c1534d2047d95b55a0dc70a7dea2f863a8c7c7c2 WatchSource:0}: Error finding container abddd4bd64b1a7fdf9591474c1534d2047d95b55a0dc70a7dea2f863a8c7c7c2: Status 404 returned error can't find the container with id abddd4bd64b1a7fdf9591474c1534d2047d95b55a0dc70a7dea2f863a8c7c7c2
May 25 11:29:13.099220 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:13.099180    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d" event={"ID":"7ea541a2-254d-48a0-b060-cb26f5180b6f","Type":"ContainerStarted","Data":"abddd4bd64b1a7fdf9591474c1534d2047d95b55a0dc70a7dea2f863a8c7c7c2"}
May 25 11:29:14.103603 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:14.103562    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d" event={"ID":"7ea541a2-254d-48a0-b060-cb26f5180b6f","Type":"ContainerStarted","Data":"12d5541890335200a57bdbc54c5655d80621b5d7e139b7dab2b97fa619a5501a"}
May 25 11:29:14.152382 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:14.151095    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-54b9655956-ftq6d" podStartSLOduration=1.609607919 podStartE2EDuration="3.151076662s" podCreationTimestamp="2026-05-25 11:29:11 +0000 UTC" firstStartedPulling="2026-05-25 11:29:12.134147715 +0000 UTC m=+411.783432688" lastFinishedPulling="2026-05-25 11:29:13.675616441 +0000 UTC m=+413.324901431" observedRunningTime="2026-05-25 11:29:14.148013224 +0000 UTC m=+413.797298221" watchObservedRunningTime="2026-05-25 11:29:14.151076662 +0000 UTC m=+413.800361653"
May 25 11:29:20.969336 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:20.969303    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"]
May 25 11:29:20.972602 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:20.972583    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:20.975325 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:20.975304    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:29:20.976329 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:20.976311    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:29:20.976329 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:20.976325    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6sjvf\""
May 25 11:29:20.980555 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:20.980526    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"]
May 25 11:29:21.124533 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:21.124503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9vhj\" (UniqueName: \"kubernetes.io/projected/464079a1-19ac-4932-b216-1ae709516add-kube-api-access-d9vhj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:21.124702 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:21.124540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:21.124702 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:21.124613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:21.225645 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:21.225609    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:21.225815 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:21.225665    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:21.225815 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:21.225710    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d9vhj\" (UniqueName: \"kubernetes.io/projected/464079a1-19ac-4932-b216-1ae709516add-kube-api-access-d9vhj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:21.226014 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:21.225992    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:21.226051 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:21.226022    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:21.234179 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:21.234153    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9vhj\" (UniqueName: \"kubernetes.io/projected/464079a1-19ac-4932-b216-1ae709516add-kube-api-access-d9vhj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:21.282323 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:21.282286    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:21.412772 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:21.411907    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"]
May 25 11:29:22.128341 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:22.128310    2578 generic.go:358] "Generic (PLEG): container finished" podID="464079a1-19ac-4932-b216-1ae709516add" containerID="e3a1b5d859c349bb7a9454caf7aba48fef9ae8640e16a520a554ac26bd3dc7dd" exitCode=0
May 25 11:29:22.128741 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:22.128378    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd" event={"ID":"464079a1-19ac-4932-b216-1ae709516add","Type":"ContainerDied","Data":"e3a1b5d859c349bb7a9454caf7aba48fef9ae8640e16a520a554ac26bd3dc7dd"}
May 25 11:29:22.128741 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:22.128400    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd" event={"ID":"464079a1-19ac-4932-b216-1ae709516add","Type":"ContainerStarted","Data":"5172916b8371a3716e16aa5896c9d87240491b33148d285cbb9dc2b4bb6ef996"}
May 25 11:29:25.139151 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:25.139111    2578 generic.go:358] "Generic (PLEG): container finished" podID="464079a1-19ac-4932-b216-1ae709516add" containerID="1aa68c328f17549dd5cdfec40e3352674a0f9463595f4b0675e3bc4364078fd9" exitCode=0
May 25 11:29:25.139597 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:25.139178    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd" event={"ID":"464079a1-19ac-4932-b216-1ae709516add","Type":"ContainerDied","Data":"1aa68c328f17549dd5cdfec40e3352674a0f9463595f4b0675e3bc4364078fd9"}
May 25 11:29:26.144731 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:26.144693    2578 generic.go:358] "Generic (PLEG): container finished" podID="464079a1-19ac-4932-b216-1ae709516add" containerID="df699f2cd440b8717cc4a449edd3497a32cd965c31a1b949c87b8b727c277215" exitCode=0
May 25 11:29:26.145125 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:26.144757    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd" event={"ID":"464079a1-19ac-4932-b216-1ae709516add","Type":"ContainerDied","Data":"df699f2cd440b8717cc4a449edd3497a32cd965c31a1b949c87b8b727c277215"}
May 25 11:29:27.274445 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:27.274417    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:27.373012 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:27.372972    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-util\") pod \"464079a1-19ac-4932-b216-1ae709516add\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") "
May 25 11:29:27.373173 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:27.373035    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-bundle\") pod \"464079a1-19ac-4932-b216-1ae709516add\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") "
May 25 11:29:27.373173 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:27.373065    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9vhj\" (UniqueName: \"kubernetes.io/projected/464079a1-19ac-4932-b216-1ae709516add-kube-api-access-d9vhj\") pod \"464079a1-19ac-4932-b216-1ae709516add\" (UID: \"464079a1-19ac-4932-b216-1ae709516add\") "
May 25 11:29:27.373446 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:27.373418    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-bundle" (OuterVolumeSpecName: "bundle") pod "464079a1-19ac-4932-b216-1ae709516add" (UID: "464079a1-19ac-4932-b216-1ae709516add"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:29:27.375309 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:27.375281    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464079a1-19ac-4932-b216-1ae709516add-kube-api-access-d9vhj" (OuterVolumeSpecName: "kube-api-access-d9vhj") pod "464079a1-19ac-4932-b216-1ae709516add" (UID: "464079a1-19ac-4932-b216-1ae709516add"). InnerVolumeSpecName "kube-api-access-d9vhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:29:27.377417 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:27.377392    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-util" (OuterVolumeSpecName: "util") pod "464079a1-19ac-4932-b216-1ae709516add" (UID: "464079a1-19ac-4932-b216-1ae709516add"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:29:27.473840 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:27.473809    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-bundle\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:29:27.473840 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:27.473840    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9vhj\" (UniqueName: \"kubernetes.io/projected/464079a1-19ac-4932-b216-1ae709516add-kube-api-access-d9vhj\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:29:27.474020 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:27.473850    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/464079a1-19ac-4932-b216-1ae709516add-util\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:29:28.152791 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:28.152747    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd" event={"ID":"464079a1-19ac-4932-b216-1ae709516add","Type":"ContainerDied","Data":"5172916b8371a3716e16aa5896c9d87240491b33148d285cbb9dc2b4bb6ef996"}
May 25 11:29:28.152791 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:28.152781    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5172916b8371a3716e16aa5896c9d87240491b33148d285cbb9dc2b4bb6ef996"
May 25 11:29:28.153017 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:28.152817    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkdwkd"
May 25 11:29:50.572568 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.572480    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"]
May 25 11:29:50.573006 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.572788    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464079a1-19ac-4932-b216-1ae709516add" containerName="util"
May 25 11:29:50.573006 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.572801    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="464079a1-19ac-4932-b216-1ae709516add" containerName="util"
May 25 11:29:50.573006 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.572816    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464079a1-19ac-4932-b216-1ae709516add" containerName="pull"
May 25 11:29:50.573006 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.572821    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="464079a1-19ac-4932-b216-1ae709516add" containerName="pull"
May 25 11:29:50.573006 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.572830    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464079a1-19ac-4932-b216-1ae709516add" containerName="extract"
May 25 11:29:50.573006 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.572835    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="464079a1-19ac-4932-b216-1ae709516add" containerName="extract"
May 25 11:29:50.573006 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.572875    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="464079a1-19ac-4932-b216-1ae709516add" containerName="extract"
May 25 11:29:50.574674 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.574657    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:50.577617 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.577596    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6sjvf\""
May 25 11:29:50.577797 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.577785    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:29:50.578771 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.578746    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:29:50.584945 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.584921    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"]
May 25 11:29:50.638568 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.638514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:50.638568 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.638579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p78tn\" (UniqueName: \"kubernetes.io/projected/4a4e6bac-eb11-4a94-8316-2b9010bda41b-kube-api-access-p78tn\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:50.638787 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.638614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:50.738944 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.738901    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p78tn\" (UniqueName: \"kubernetes.io/projected/4a4e6bac-eb11-4a94-8316-2b9010bda41b-kube-api-access-p78tn\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:50.739119 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.738966    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:50.739119 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.739024    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:50.739371 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.739351    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:50.739406 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.739383    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:50.747755 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.747728    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78tn\" (UniqueName: \"kubernetes.io/projected/4a4e6bac-eb11-4a94-8316-2b9010bda41b-kube-api-access-p78tn\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:50.884026 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:50.883932    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:51.014586 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:51.014557    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"]
May 25 11:29:51.018244 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:29:51.018213    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4e6bac_eb11_4a94_8316_2b9010bda41b.slice/crio-148cd1b4a56aff61c6c93dc4efd20c9f93a75c3f907cd8e787d56027793c5852 WatchSource:0}: Error finding container 148cd1b4a56aff61c6c93dc4efd20c9f93a75c3f907cd8e787d56027793c5852: Status 404 returned error can't find the container with id 148cd1b4a56aff61c6c93dc4efd20c9f93a75c3f907cd8e787d56027793c5852
May 25 11:29:51.227338 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:51.227303    2578 generic.go:358] "Generic (PLEG): container finished" podID="4a4e6bac-eb11-4a94-8316-2b9010bda41b" containerID="f7d6268c2f2a4aad4c155e372014082c6f866a081809f264516dbb62ce861202" exitCode=0
May 25 11:29:51.227523 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:51.227348    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr" event={"ID":"4a4e6bac-eb11-4a94-8316-2b9010bda41b","Type":"ContainerDied","Data":"f7d6268c2f2a4aad4c155e372014082c6f866a081809f264516dbb62ce861202"}
May 25 11:29:51.227523 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:51.227388    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr" event={"ID":"4a4e6bac-eb11-4a94-8316-2b9010bda41b","Type":"ContainerStarted","Data":"148cd1b4a56aff61c6c93dc4efd20c9f93a75c3f907cd8e787d56027793c5852"}
May 25 11:29:52.231860 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:52.231826    2578 generic.go:358] "Generic (PLEG): container finished" podID="4a4e6bac-eb11-4a94-8316-2b9010bda41b" containerID="71870a15a397859bfd468bbcc2bb47fb68116b9a5a1eec0316c9c77727301012" exitCode=0
May 25 11:29:52.232257 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:52.231894    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr" event={"ID":"4a4e6bac-eb11-4a94-8316-2b9010bda41b","Type":"ContainerDied","Data":"71870a15a397859bfd468bbcc2bb47fb68116b9a5a1eec0316c9c77727301012"}
May 25 11:29:53.236717 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:53.236685    2578 generic.go:358] "Generic (PLEG): container finished" podID="4a4e6bac-eb11-4a94-8316-2b9010bda41b" containerID="fd4e41773972ea7fd623f25037472dd9711a35c2aa970ca7fa84671bbea85cd5" exitCode=0
May 25 11:29:53.237095 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:53.236760    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr" event={"ID":"4a4e6bac-eb11-4a94-8316-2b9010bda41b","Type":"ContainerDied","Data":"fd4e41773972ea7fd623f25037472dd9711a35c2aa970ca7fa84671bbea85cd5"}
May 25 11:29:54.357594 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:54.357567    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:29:54.465360 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:54.465326    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-bundle\") pod \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") "
May 25 11:29:54.465564 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:54.465374    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p78tn\" (UniqueName: \"kubernetes.io/projected/4a4e6bac-eb11-4a94-8316-2b9010bda41b-kube-api-access-p78tn\") pod \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") "
May 25 11:29:54.465564 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:54.465403    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-util\") pod \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\" (UID: \"4a4e6bac-eb11-4a94-8316-2b9010bda41b\") "
May 25 11:29:54.466251 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:54.466218    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-bundle" (OuterVolumeSpecName: "bundle") pod "4a4e6bac-eb11-4a94-8316-2b9010bda41b" (UID: "4a4e6bac-eb11-4a94-8316-2b9010bda41b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:29:54.467669 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:54.467645    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4e6bac-eb11-4a94-8316-2b9010bda41b-kube-api-access-p78tn" (OuterVolumeSpecName: "kube-api-access-p78tn") pod "4a4e6bac-eb11-4a94-8316-2b9010bda41b" (UID: "4a4e6bac-eb11-4a94-8316-2b9010bda41b"). InnerVolumeSpecName "kube-api-access-p78tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:29:54.471078 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:54.471055    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-util" (OuterVolumeSpecName: "util") pod "4a4e6bac-eb11-4a94-8316-2b9010bda41b" (UID: "4a4e6bac-eb11-4a94-8316-2b9010bda41b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:29:54.566766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:54.566678    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-bundle\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:29:54.566766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:54.566710    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p78tn\" (UniqueName: \"kubernetes.io/projected/4a4e6bac-eb11-4a94-8316-2b9010bda41b-kube-api-access-p78tn\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:29:54.566766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:54.566721    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a4e6bac-eb11-4a94-8316-2b9010bda41b-util\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:29:55.243475 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:55.243422    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr" event={"ID":"4a4e6bac-eb11-4a94-8316-2b9010bda41b","Type":"ContainerDied","Data":"148cd1b4a56aff61c6c93dc4efd20c9f93a75c3f907cd8e787d56027793c5852"}
May 25 11:29:55.243475 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:55.243473    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="148cd1b4a56aff61c6c93dc4efd20c9f93a75c3f907cd8e787d56027793c5852"
May 25 11:29:55.243672 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:29:55.243484    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835l92vr"
May 25 11:30:00.864829 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.864790    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"]
May 25 11:30:00.865294 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.865131    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4e6bac-eb11-4a94-8316-2b9010bda41b" containerName="util"
May 25 11:30:00.865294 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.865149    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4e6bac-eb11-4a94-8316-2b9010bda41b" containerName="util"
May 25 11:30:00.865294 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.865175    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4e6bac-eb11-4a94-8316-2b9010bda41b" containerName="extract"
May 25 11:30:00.865294 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.865185    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4e6bac-eb11-4a94-8316-2b9010bda41b" containerName="extract"
May 25 11:30:00.865294 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.865196    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4e6bac-eb11-4a94-8316-2b9010bda41b" containerName="pull"
May 25 11:30:00.865294 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.865204    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4e6bac-eb11-4a94-8316-2b9010bda41b" containerName="pull"
May 25 11:30:00.865294 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.865280    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="4a4e6bac-eb11-4a94-8316-2b9010bda41b" containerName="extract"
May 25 11:30:00.867560 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.867538    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:00.870992 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.870945    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:30:00.871768 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.871751    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6sjvf\""
May 25 11:30:00.871888 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.871758    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:30:00.884702 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.884657    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"]
May 25 11:30:00.907543 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.907508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:00.907673 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.907555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdn2j\" (UniqueName: \"kubernetes.io/projected/492c57b9-57f6-4d6e-8c05-337b082ed089-kube-api-access-gdn2j\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:00.907673 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:00.907581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:01.008069 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:01.008037    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:01.008223 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:01.008092    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:01.008223 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:01.008135    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gdn2j\" (UniqueName: \"kubernetes.io/projected/492c57b9-57f6-4d6e-8c05-337b082ed089-kube-api-access-gdn2j\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:01.008431 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:01.008409    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-bundle\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:01.008531 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:01.008512    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-util\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:01.019260 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:01.019231    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdn2j\" (UniqueName: \"kubernetes.io/projected/492c57b9-57f6-4d6e-8c05-337b082ed089-kube-api-access-gdn2j\") pod \"c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") " pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:01.176365 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:01.176279    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:01.341589 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:01.341567    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"]
May 25 11:30:01.344282 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:30:01.344243    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492c57b9_57f6_4d6e_8c05_337b082ed089.slice/crio-8e0227108d497f33b5f875b50c230d076dc74f5d532f86640fb8d39efc9e2275 WatchSource:0}: Error finding container 8e0227108d497f33b5f875b50c230d076dc74f5d532f86640fb8d39efc9e2275: Status 404 returned error can't find the container with id 8e0227108d497f33b5f875b50c230d076dc74f5d532f86640fb8d39efc9e2275
May 25 11:30:02.138433 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.138400    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"]
May 25 11:30:02.140298 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.140282    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"
May 25 11:30:02.143355 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.143318    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"kube-root-ca.crt\""
May 25 11:30:02.143631 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.143612    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators\"/\"servicemesh-operator3-dockercfg-5555r\""
May 25 11:30:02.143945 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.143931    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"openshift-service-ca.crt\""
May 25 11:30:02.168120 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.168091    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"]
May 25 11:30:02.217513 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.217452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/5c9a1fa0-ada3-4ba4-8b05-932f85bbc004-operator-config\") pod \"servicemesh-operator3-55f49c5f94-cpbsx\" (UID: \"5c9a1fa0-ada3-4ba4-8b05-932f85bbc004\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"
May 25 11:30:02.217513 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.217514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhx2\" (UniqueName: \"kubernetes.io/projected/5c9a1fa0-ada3-4ba4-8b05-932f85bbc004-kube-api-access-bfhx2\") pod \"servicemesh-operator3-55f49c5f94-cpbsx\" (UID: \"5c9a1fa0-ada3-4ba4-8b05-932f85bbc004\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"
May 25 11:30:02.267940 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.267905    2578 generic.go:358] "Generic (PLEG): container finished" podID="492c57b9-57f6-4d6e-8c05-337b082ed089" containerID="d434b6e3343f92f5c308b669f4ffd60b5a33963e14c9520a318d9ebb1f6e0a5a" exitCode=0
May 25 11:30:02.267940 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.267942    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g" event={"ID":"492c57b9-57f6-4d6e-8c05-337b082ed089","Type":"ContainerDied","Data":"d434b6e3343f92f5c308b669f4ffd60b5a33963e14c9520a318d9ebb1f6e0a5a"}
May 25 11:30:02.268151 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.267965    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g" event={"ID":"492c57b9-57f6-4d6e-8c05-337b082ed089","Type":"ContainerStarted","Data":"8e0227108d497f33b5f875b50c230d076dc74f5d532f86640fb8d39efc9e2275"}
May 25 11:30:02.318130 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.318039    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/5c9a1fa0-ada3-4ba4-8b05-932f85bbc004-operator-config\") pod \"servicemesh-operator3-55f49c5f94-cpbsx\" (UID: \"5c9a1fa0-ada3-4ba4-8b05-932f85bbc004\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"
May 25 11:30:02.318130 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.318094    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhx2\" (UniqueName: \"kubernetes.io/projected/5c9a1fa0-ada3-4ba4-8b05-932f85bbc004-kube-api-access-bfhx2\") pod \"servicemesh-operator3-55f49c5f94-cpbsx\" (UID: \"5c9a1fa0-ada3-4ba4-8b05-932f85bbc004\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"
May 25 11:30:02.320730 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.320700    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/5c9a1fa0-ada3-4ba4-8b05-932f85bbc004-operator-config\") pod \"servicemesh-operator3-55f49c5f94-cpbsx\" (UID: \"5c9a1fa0-ada3-4ba4-8b05-932f85bbc004\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"
May 25 11:30:02.341942 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.341908    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhx2\" (UniqueName: \"kubernetes.io/projected/5c9a1fa0-ada3-4ba4-8b05-932f85bbc004-kube-api-access-bfhx2\") pod \"servicemesh-operator3-55f49c5f94-cpbsx\" (UID: \"5c9a1fa0-ada3-4ba4-8b05-932f85bbc004\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"
May 25 11:30:02.449237 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.449203    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"
May 25 11:30:02.581953 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:02.581928    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"]
May 25 11:30:02.584369 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:30:02.584338    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9a1fa0_ada3_4ba4_8b05_932f85bbc004.slice/crio-2904acc8d4b4910ce940086128036c42e3e8d8ee80966c4e1bdc38bca4a51328 WatchSource:0}: Error finding container 2904acc8d4b4910ce940086128036c42e3e8d8ee80966c4e1bdc38bca4a51328: Status 404 returned error can't find the container with id 2904acc8d4b4910ce940086128036c42e3e8d8ee80966c4e1bdc38bca4a51328
May 25 11:30:03.272213 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:03.272173    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx" event={"ID":"5c9a1fa0-ada3-4ba4-8b05-932f85bbc004","Type":"ContainerStarted","Data":"2904acc8d4b4910ce940086128036c42e3e8d8ee80966c4e1bdc38bca4a51328"}
May 25 11:30:05.284709 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:05.284667    2578 generic.go:358] "Generic (PLEG): container finished" podID="492c57b9-57f6-4d6e-8c05-337b082ed089" containerID="b13844ceb83031b435a6a9d70e801a462b7bfcaf6e934fae81ac053f9cbdd99f" exitCode=0
May 25 11:30:05.285173 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:05.284750    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g" event={"ID":"492c57b9-57f6-4d6e-8c05-337b082ed089","Type":"ContainerDied","Data":"b13844ceb83031b435a6a9d70e801a462b7bfcaf6e934fae81ac053f9cbdd99f"}
May 25 11:30:06.289941 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:06.289895    2578 generic.go:358] "Generic (PLEG): container finished" podID="492c57b9-57f6-4d6e-8c05-337b082ed089" containerID="209a949099df2b6ac5f25b9a78921c4bfb5761fec0884a579dfd805b3213ba90" exitCode=0
May 25 11:30:06.290305 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:06.289960    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g" event={"ID":"492c57b9-57f6-4d6e-8c05-337b082ed089","Type":"ContainerDied","Data":"209a949099df2b6ac5f25b9a78921c4bfb5761fec0884a579dfd805b3213ba90"}
May 25 11:30:07.419794 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:07.419769    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:07.561634 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:07.561553    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdn2j\" (UniqueName: \"kubernetes.io/projected/492c57b9-57f6-4d6e-8c05-337b082ed089-kube-api-access-gdn2j\") pod \"492c57b9-57f6-4d6e-8c05-337b082ed089\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") "
May 25 11:30:07.561634 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:07.561603    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-bundle\") pod \"492c57b9-57f6-4d6e-8c05-337b082ed089\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") "
May 25 11:30:07.561808 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:07.561641    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-util\") pod \"492c57b9-57f6-4d6e-8c05-337b082ed089\" (UID: \"492c57b9-57f6-4d6e-8c05-337b082ed089\") "
May 25 11:30:07.562580 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:07.562544    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-bundle" (OuterVolumeSpecName: "bundle") pod "492c57b9-57f6-4d6e-8c05-337b082ed089" (UID: "492c57b9-57f6-4d6e-8c05-337b082ed089"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:07.563851 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:07.563831    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492c57b9-57f6-4d6e-8c05-337b082ed089-kube-api-access-gdn2j" (OuterVolumeSpecName: "kube-api-access-gdn2j") pod "492c57b9-57f6-4d6e-8c05-337b082ed089" (UID: "492c57b9-57f6-4d6e-8c05-337b082ed089"). InnerVolumeSpecName "kube-api-access-gdn2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:30:07.566197 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:07.566172    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-util" (OuterVolumeSpecName: "util") pod "492c57b9-57f6-4d6e-8c05-337b082ed089" (UID: "492c57b9-57f6-4d6e-8c05-337b082ed089"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:07.662499 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:07.662443    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-util\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:07.662499 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:07.662494    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdn2j\" (UniqueName: \"kubernetes.io/projected/492c57b9-57f6-4d6e-8c05-337b082ed089-kube-api-access-gdn2j\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:07.662499 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:07.662506    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/492c57b9-57f6-4d6e-8c05-337b082ed089-bundle\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:08.298140 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.298109    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g"
May 25 11:30:08.298319 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.298111    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c34dbd22d61532abdd91a8be490d70c9b1095425e8ccf73b542d2e762amrq7g" event={"ID":"492c57b9-57f6-4d6e-8c05-337b082ed089","Type":"ContainerDied","Data":"8e0227108d497f33b5f875b50c230d076dc74f5d532f86640fb8d39efc9e2275"}
May 25 11:30:08.298319 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.298231    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e0227108d497f33b5f875b50c230d076dc74f5d532f86640fb8d39efc9e2275"
May 25 11:30:08.776530 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.776497    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"]
May 25 11:30:08.776975 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.776777    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492c57b9-57f6-4d6e-8c05-337b082ed089" containerName="extract"
May 25 11:30:08.776975 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.776792    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="492c57b9-57f6-4d6e-8c05-337b082ed089" containerName="extract"
May 25 11:30:08.776975 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.776803    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492c57b9-57f6-4d6e-8c05-337b082ed089" containerName="pull"
May 25 11:30:08.776975 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.776810    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="492c57b9-57f6-4d6e-8c05-337b082ed089" containerName="pull"
May 25 11:30:08.776975 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.776817    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492c57b9-57f6-4d6e-8c05-337b082ed089" containerName="util"
May 25 11:30:08.776975 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.776824    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="492c57b9-57f6-4d6e-8c05-337b082ed089" containerName="util"
May 25 11:30:08.776975 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.776881    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="492c57b9-57f6-4d6e-8c05-337b082ed089" containerName="extract"
May 25 11:30:08.779660 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.779636    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.783654 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.783628    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 25 11:30:08.783781 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.783731    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hssbk\""
May 25 11:30:08.784977 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.784936    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 25 11:30:08.784977 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.784962    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 25 11:30:08.785504 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.785484    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 25 11:30:08.787573 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.787338    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 25 11:30:08.804273 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.804250    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"]
May 25 11:30:08.872746 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.872704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qw2x\" (UniqueName: \"kubernetes.io/projected/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-kube-api-access-4qw2x\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.872932 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.872764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-cert\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.872932 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.872814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-metrics-cert\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.872932 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.872842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-manager-config\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.973979 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.973942    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4qw2x\" (UniqueName: \"kubernetes.io/projected/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-kube-api-access-4qw2x\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.974157 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.974016    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-cert\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.974157 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.974058    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-metrics-cert\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.974157 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.974094    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-manager-config\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.974881 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.974856    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-manager-config\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.976791 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.976764    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-metrics-cert\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.976899 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.976830    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-cert\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:08.986589 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:08.986567    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qw2x\" (UniqueName: \"kubernetes.io/projected/63aa5fd3-ccd9-497f-90c8-e023f06b7dfe-kube-api-access-4qw2x\") pod \"lws-controller-manager-8646484d7f-glghm\" (UID: \"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe\") " pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:09.090582 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:09.090550    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:09.250908 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:09.250879    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"]
May 25 11:30:09.254109 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:30:09.254064    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63aa5fd3_ccd9_497f_90c8_e023f06b7dfe.slice/crio-29c3c1353ea8023286438fc04dd88c7b112c98d11a9e6f5d64cb69300642965f WatchSource:0}: Error finding container 29c3c1353ea8023286438fc04dd88c7b112c98d11a9e6f5d64cb69300642965f: Status 404 returned error can't find the container with id 29c3c1353ea8023286438fc04dd88c7b112c98d11a9e6f5d64cb69300642965f
May 25 11:30:09.304752 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:09.304639    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx" event={"ID":"5c9a1fa0-ada3-4ba4-8b05-932f85bbc004","Type":"ContainerStarted","Data":"a7bcbfb27de88ceb1981a031da1e0c321407ac72100357147d8c5f0c0984bc30"}
May 25 11:30:09.304920 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:09.304879    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"
May 25 11:30:09.306010 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:09.305965    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm" event={"ID":"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe","Type":"ContainerStarted","Data":"29c3c1353ea8023286438fc04dd88c7b112c98d11a9e6f5d64cb69300642965f"}
May 25 11:30:09.329749 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:09.329698    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx" podStartSLOduration=0.891494095 podStartE2EDuration="7.329682352s" podCreationTimestamp="2026-05-25 11:30:02 +0000 UTC" firstStartedPulling="2026-05-25 11:30:02.586851649 +0000 UTC m=+462.236136622" lastFinishedPulling="2026-05-25 11:30:09.025039893 +0000 UTC m=+468.674324879" observedRunningTime="2026-05-25 11:30:09.32658763 +0000 UTC m=+468.975872638" watchObservedRunningTime="2026-05-25 11:30:09.329682352 +0000 UTC m=+468.978967348"
May 25 11:30:20.310850 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:20.310817    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-cpbsx"
May 25 11:30:20.342486 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:20.342427    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm" event={"ID":"63aa5fd3-ccd9-497f-90c8-e023f06b7dfe","Type":"ContainerStarted","Data":"d13ddf1bf8293b7ffa2de70ecd7be02b10e82f4f092818625015c4bf4e9d3463"}
May 25 11:30:20.342671 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:20.342516    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:20.360075 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:20.360021    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm" podStartSLOduration=1.6596450520000001 podStartE2EDuration="12.360006977s" podCreationTimestamp="2026-05-25 11:30:08 +0000 UTC" firstStartedPulling="2026-05-25 11:30:09.256283752 +0000 UTC m=+468.905568737" lastFinishedPulling="2026-05-25 11:30:19.956645675 +0000 UTC m=+479.605930662" observedRunningTime="2026-05-25 11:30:20.359335686 +0000 UTC m=+480.008620682" watchObservedRunningTime="2026-05-25 11:30:20.360006977 +0000 UTC m=+480.009291971"
May 25 11:30:31.347980 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:31.347948    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8646484d7f-glghm"
May 25 11:30:48.265665 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.265625    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"]
May 25 11:30:48.267697 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.267680    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:48.270745 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.270721    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6sjvf\""
May 25 11:30:48.271738 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.271719    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:30:48.271815 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.271725    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:30:48.276490 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.276449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:48.276695 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.276671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:48.276801 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.276774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t769w\" (UniqueName: \"kubernetes.io/projected/a71d7972-772e-40a4-bc14-8c08acc8bd6f-kube-api-access-t769w\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:48.277257 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.277238    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"]
May 25 11:30:48.377703 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.377667    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t769w\" (UniqueName: \"kubernetes.io/projected/a71d7972-772e-40a4-bc14-8c08acc8bd6f-kube-api-access-t769w\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:48.377887 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.377718    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:48.377887 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.377781    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:48.378154 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.378131    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-util\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:48.378235 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.378131    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-bundle\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:48.385752 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.385722    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"]
May 25 11:30:48.388107 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.388087    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:48.389897 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.389873    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t769w\" (UniqueName: \"kubernetes.io/projected/a71d7972-772e-40a4-bc14-8c08acc8bd6f-kube-api-access-t769w\") pod \"b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") " pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:48.417918 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.417879    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"]
May 25 11:30:48.478487 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.478430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:48.478487 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.478487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:48.478714 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.478533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zwdp\" (UniqueName: \"kubernetes.io/projected/ca62d57d-2c20-40bb-997c-8e32a776ecb8-kube-api-access-7zwdp\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:48.552631 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.552554    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"]
May 25 11:30:48.554626 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.554607    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:48.566008 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.565983    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"]
May 25 11:30:48.577237 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.577213    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:48.579191 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.579171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cww6l\" (UniqueName: \"kubernetes.io/projected/5d230960-8a85-4e7b-9131-d70420b5d609-kube-api-access-cww6l\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:48.579251 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.579209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:48.579288 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.579271    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:48.579333 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.579313    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:48.579374 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.579353    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7zwdp\" (UniqueName: \"kubernetes.io/projected/ca62d57d-2c20-40bb-997c-8e32a776ecb8-kube-api-access-7zwdp\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:48.579415 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.579387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:48.579621 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.579604    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:48.579662 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.579634    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:48.589539 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.589517    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zwdp\" (UniqueName: \"kubernetes.io/projected/ca62d57d-2c20-40bb-997c-8e32a776ecb8-kube-api-access-7zwdp\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:48.616547 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.616511    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"]
May 25 11:30:48.619214 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.619194    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:48.629555 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.629516    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"]
May 25 11:30:48.680253 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.680185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:48.680253 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.680220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nppq8\" (UniqueName: \"kubernetes.io/projected/3bd7bda8-1bcc-4459-bf6c-35623e54139d-kube-api-access-nppq8\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:48.680253 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.680256    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cww6l\" (UniqueName: \"kubernetes.io/projected/5d230960-8a85-4e7b-9131-d70420b5d609-kube-api-access-cww6l\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:48.680253 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.680287    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:48.680253 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.680310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:48.680253 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.680367    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:48.680900 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.680759    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:48.680900 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.680773    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:48.689206 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.689177    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cww6l\" (UniqueName: \"kubernetes.io/projected/5d230960-8a85-4e7b-9131-d70420b5d609-kube-api-access-cww6l\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:48.702752 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.702727    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:48.707393 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.707372    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"]
May 25 11:30:48.710107 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:30:48.710078    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71d7972_772e_40a4_bc14_8c08acc8bd6f.slice/crio-ca04c4d014733791a32e1e0b24fbc87da5dc2df783362a616dfc656c972c56f1 WatchSource:0}: Error finding container ca04c4d014733791a32e1e0b24fbc87da5dc2df783362a616dfc656c972c56f1: Status 404 returned error can't find the container with id ca04c4d014733791a32e1e0b24fbc87da5dc2df783362a616dfc656c972c56f1
May 25 11:30:48.781066 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.781041    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:48.781194 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.781075    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nppq8\" (UniqueName: \"kubernetes.io/projected/3bd7bda8-1bcc-4459-bf6c-35623e54139d-kube-api-access-nppq8\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:48.781194 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.781133    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:48.781527 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.781501    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:48.781527 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.781518    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:48.793355 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.793322    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nppq8\" (UniqueName: \"kubernetes.io/projected/3bd7bda8-1bcc-4459-bf6c-35623e54139d-kube-api-access-nppq8\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:48.838102 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.838078    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"]
May 25 11:30:48.863288 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.863254    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:48.876619 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:30:48.876589    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca62d57d_2c20_40bb_997c_8e32a776ecb8.slice/crio-3d8f06bf241daa822f05cb78e1e84fa455a130752145210910666545959a498f WatchSource:0}: Error finding container 3d8f06bf241daa822f05cb78e1e84fa455a130752145210910666545959a498f: Status 404 returned error can't find the container with id 3d8f06bf241daa822f05cb78e1e84fa455a130752145210910666545959a498f
May 25 11:30:48.931274 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:48.931247    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:49.004849 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.002217    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"]
May 25 11:30:49.039528 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:30:49.039493    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d230960_8a85_4e7b_9131_d70420b5d609.slice/crio-9637b92f10a21ede9c1c66bb42fcb67a66f4728d8c294db0d2fc65612a0e752b WatchSource:0}: Error finding container 9637b92f10a21ede9c1c66bb42fcb67a66f4728d8c294db0d2fc65612a0e752b: Status 404 returned error can't find the container with id 9637b92f10a21ede9c1c66bb42fcb67a66f4728d8c294db0d2fc65612a0e752b
May 25 11:30:49.075689 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.075665    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"]
May 25 11:30:49.077767 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:30:49.077742    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd7bda8_1bcc_4459_bf6c_35623e54139d.slice/crio-0d6f836e3dbc56ae082944e3a83110ac42fbd58874079b2e36aa0d4fb9b3b24b WatchSource:0}: Error finding container 0d6f836e3dbc56ae082944e3a83110ac42fbd58874079b2e36aa0d4fb9b3b24b: Status 404 returned error can't find the container with id 0d6f836e3dbc56ae082944e3a83110ac42fbd58874079b2e36aa0d4fb9b3b24b
May 25 11:30:49.437566 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.437469    2578 generic.go:358] "Generic (PLEG): container finished" podID="5d230960-8a85-4e7b-9131-d70420b5d609" containerID="53313ab1e94b81f9425063efc616ac07eafe1ee0365bb93a2d94a41de0999b24" exitCode=0
May 25 11:30:49.437566 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.437494    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk" event={"ID":"5d230960-8a85-4e7b-9131-d70420b5d609","Type":"ContainerDied","Data":"53313ab1e94b81f9425063efc616ac07eafe1ee0365bb93a2d94a41de0999b24"}
May 25 11:30:49.437566 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.437536    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk" event={"ID":"5d230960-8a85-4e7b-9131-d70420b5d609","Type":"ContainerStarted","Data":"9637b92f10a21ede9c1c66bb42fcb67a66f4728d8c294db0d2fc65612a0e752b"}
May 25 11:30:49.439122 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.439098    2578 generic.go:358] "Generic (PLEG): container finished" podID="a71d7972-772e-40a4-bc14-8c08acc8bd6f" containerID="9cc14ac92aee4460f68a1fbaeb9df0b159071c34936da094565c496faf9c19f8" exitCode=0
May 25 11:30:49.439221 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.439169    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj" event={"ID":"a71d7972-772e-40a4-bc14-8c08acc8bd6f","Type":"ContainerDied","Data":"9cc14ac92aee4460f68a1fbaeb9df0b159071c34936da094565c496faf9c19f8"}
May 25 11:30:49.439221 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.439205    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj" event={"ID":"a71d7972-772e-40a4-bc14-8c08acc8bd6f","Type":"ContainerStarted","Data":"ca04c4d014733791a32e1e0b24fbc87da5dc2df783362a616dfc656c972c56f1"}
May 25 11:30:49.440645 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.440626    2578 generic.go:358] "Generic (PLEG): container finished" podID="ca62d57d-2c20-40bb-997c-8e32a776ecb8" containerID="3d2a8251220071b5dc9b4e972dd096dd0d956ee06b6d1a26d5e60b93631d6e7c" exitCode=0
May 25 11:30:49.440742 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.440681    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc" event={"ID":"ca62d57d-2c20-40bb-997c-8e32a776ecb8","Type":"ContainerDied","Data":"3d2a8251220071b5dc9b4e972dd096dd0d956ee06b6d1a26d5e60b93631d6e7c"}
May 25 11:30:49.440742 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.440709    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc" event={"ID":"ca62d57d-2c20-40bb-997c-8e32a776ecb8","Type":"ContainerStarted","Data":"3d8f06bf241daa822f05cb78e1e84fa455a130752145210910666545959a498f"}
May 25 11:30:49.442144 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.442119    2578 generic.go:358] "Generic (PLEG): container finished" podID="3bd7bda8-1bcc-4459-bf6c-35623e54139d" containerID="44648f07c70fab599e132a964ae473cc1658d1b87e492d2226da4addfaac3c82" exitCode=0
May 25 11:30:49.442264 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.442189    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc" event={"ID":"3bd7bda8-1bcc-4459-bf6c-35623e54139d","Type":"ContainerDied","Data":"44648f07c70fab599e132a964ae473cc1658d1b87e492d2226da4addfaac3c82"}
May 25 11:30:49.442264 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:49.442207    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc" event={"ID":"3bd7bda8-1bcc-4459-bf6c-35623e54139d","Type":"ContainerStarted","Data":"0d6f836e3dbc56ae082944e3a83110ac42fbd58874079b2e36aa0d4fb9b3b24b"}
May 25 11:30:50.447543 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:50.447439    2578 generic.go:358] "Generic (PLEG): container finished" podID="a71d7972-772e-40a4-bc14-8c08acc8bd6f" containerID="6b047ce111e48304213dcdcdcb7558fcfca8557c23a97975e0b833e32db0dcd7" exitCode=0
May 25 11:30:50.447543 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:50.447521    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj" event={"ID":"a71d7972-772e-40a4-bc14-8c08acc8bd6f","Type":"ContainerDied","Data":"6b047ce111e48304213dcdcdcb7558fcfca8557c23a97975e0b833e32db0dcd7"}
May 25 11:30:50.449226 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:50.449203    2578 generic.go:358] "Generic (PLEG): container finished" podID="3bd7bda8-1bcc-4459-bf6c-35623e54139d" containerID="bf125659c92e31306b669885f78bc5f9f7f780274d4fa765f542f3668d5b63db" exitCode=0
May 25 11:30:50.449325 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:50.449268    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc" event={"ID":"3bd7bda8-1bcc-4459-bf6c-35623e54139d","Type":"ContainerDied","Data":"bf125659c92e31306b669885f78bc5f9f7f780274d4fa765f542f3668d5b63db"}
May 25 11:30:51.453994 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:51.453958    2578 generic.go:358] "Generic (PLEG): container finished" podID="5d230960-8a85-4e7b-9131-d70420b5d609" containerID="b8682922dbcf4576a8878da3fc69f311ec182809adf27320420c33999e122a4f" exitCode=0
May 25 11:30:51.454438 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:51.454046    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk" event={"ID":"5d230960-8a85-4e7b-9131-d70420b5d609","Type":"ContainerDied","Data":"b8682922dbcf4576a8878da3fc69f311ec182809adf27320420c33999e122a4f"}
May 25 11:30:51.456021 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:51.456001    2578 generic.go:358] "Generic (PLEG): container finished" podID="a71d7972-772e-40a4-bc14-8c08acc8bd6f" containerID="6fd058bc0eb6792d77ce145f6c89ff25826add01ade79f4e5763c747c881e074" exitCode=0
May 25 11:30:51.456106 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:51.456084    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj" event={"ID":"a71d7972-772e-40a4-bc14-8c08acc8bd6f","Type":"ContainerDied","Data":"6fd058bc0eb6792d77ce145f6c89ff25826add01ade79f4e5763c747c881e074"}
May 25 11:30:51.457631 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:51.457606    2578 generic.go:358] "Generic (PLEG): container finished" podID="ca62d57d-2c20-40bb-997c-8e32a776ecb8" containerID="7ae6100f23fb185abccc5b3e227268b6b5855aa511e51e157da34d08f9687e8d" exitCode=0
May 25 11:30:51.457735 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:51.457680    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc" event={"ID":"ca62d57d-2c20-40bb-997c-8e32a776ecb8","Type":"ContainerDied","Data":"7ae6100f23fb185abccc5b3e227268b6b5855aa511e51e157da34d08f9687e8d"}
May 25 11:30:51.459583 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:51.459560    2578 generic.go:358] "Generic (PLEG): container finished" podID="3bd7bda8-1bcc-4459-bf6c-35623e54139d" containerID="6e21a58d2980d879b33189179978517f6ecf6ed1436528ac337fc248c70fa5fb" exitCode=0
May 25 11:30:51.459677 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:51.459641    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc" event={"ID":"3bd7bda8-1bcc-4459-bf6c-35623e54139d","Type":"ContainerDied","Data":"6e21a58d2980d879b33189179978517f6ecf6ed1436528ac337fc248c70fa5fb"}
May 25 11:30:52.465362 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.465327    2578 generic.go:358] "Generic (PLEG): container finished" podID="ca62d57d-2c20-40bb-997c-8e32a776ecb8" containerID="8b15849c27e379aa40cb14d469c2db824024ef13ba397d6684c03df17694296d" exitCode=0
May 25 11:30:52.465852 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.465376    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc" event={"ID":"ca62d57d-2c20-40bb-997c-8e32a776ecb8","Type":"ContainerDied","Data":"8b15849c27e379aa40cb14d469c2db824024ef13ba397d6684c03df17694296d"}
May 25 11:30:52.467364 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.467340    2578 generic.go:358] "Generic (PLEG): container finished" podID="5d230960-8a85-4e7b-9131-d70420b5d609" containerID="7bb00e35094ba156f69a63758b59fa48fde8e47f9f4c9afa512c908ec9cbce78" exitCode=0
May 25 11:30:52.467523 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.467411    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk" event={"ID":"5d230960-8a85-4e7b-9131-d70420b5d609","Type":"ContainerDied","Data":"7bb00e35094ba156f69a63758b59fa48fde8e47f9f4c9afa512c908ec9cbce78"}
May 25 11:30:52.631822 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.631798    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:52.634834 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.634814    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:52.715724 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.715644    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nppq8\" (UniqueName: \"kubernetes.io/projected/3bd7bda8-1bcc-4459-bf6c-35623e54139d-kube-api-access-nppq8\") pod \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") "
May 25 11:30:52.715724 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.715680    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-util\") pod \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") "
May 25 11:30:52.715931 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.715751    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-bundle\") pod \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\" (UID: \"3bd7bda8-1bcc-4459-bf6c-35623e54139d\") "
May 25 11:30:52.715931 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.715800    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-bundle\") pod \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") "
May 25 11:30:52.715931 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.715825    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-util\") pod \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") "
May 25 11:30:52.715931 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.715860    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t769w\" (UniqueName: \"kubernetes.io/projected/a71d7972-772e-40a4-bc14-8c08acc8bd6f-kube-api-access-t769w\") pod \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\" (UID: \"a71d7972-772e-40a4-bc14-8c08acc8bd6f\") "
May 25 11:30:52.716350 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.716315    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-bundle" (OuterVolumeSpecName: "bundle") pod "3bd7bda8-1bcc-4459-bf6c-35623e54139d" (UID: "3bd7bda8-1bcc-4459-bf6c-35623e54139d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:52.716603 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.716569    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-bundle" (OuterVolumeSpecName: "bundle") pod "a71d7972-772e-40a4-bc14-8c08acc8bd6f" (UID: "a71d7972-772e-40a4-bc14-8c08acc8bd6f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:52.718000 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.717975    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd7bda8-1bcc-4459-bf6c-35623e54139d-kube-api-access-nppq8" (OuterVolumeSpecName: "kube-api-access-nppq8") pod "3bd7bda8-1bcc-4459-bf6c-35623e54139d" (UID: "3bd7bda8-1bcc-4459-bf6c-35623e54139d"). InnerVolumeSpecName "kube-api-access-nppq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:30:52.718141 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.718118    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71d7972-772e-40a4-bc14-8c08acc8bd6f-kube-api-access-t769w" (OuterVolumeSpecName: "kube-api-access-t769w") pod "a71d7972-772e-40a4-bc14-8c08acc8bd6f" (UID: "a71d7972-772e-40a4-bc14-8c08acc8bd6f"). InnerVolumeSpecName "kube-api-access-t769w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:30:52.724905 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.724870    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-util" (OuterVolumeSpecName: "util") pod "a71d7972-772e-40a4-bc14-8c08acc8bd6f" (UID: "a71d7972-772e-40a4-bc14-8c08acc8bd6f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:52.724999 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.724944    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-util" (OuterVolumeSpecName: "util") pod "3bd7bda8-1bcc-4459-bf6c-35623e54139d" (UID: "3bd7bda8-1bcc-4459-bf6c-35623e54139d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:52.816836 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.816800    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-bundle\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:52.816836 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.816830    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a71d7972-772e-40a4-bc14-8c08acc8bd6f-util\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:52.816836 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.816843    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t769w\" (UniqueName: \"kubernetes.io/projected/a71d7972-772e-40a4-bc14-8c08acc8bd6f-kube-api-access-t769w\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:52.817027 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.816853    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nppq8\" (UniqueName: \"kubernetes.io/projected/3bd7bda8-1bcc-4459-bf6c-35623e54139d-kube-api-access-nppq8\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:52.817027 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.816862    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-util\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:52.817027 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:52.816871    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bd7bda8-1bcc-4459-bf6c-35623e54139d-bundle\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:53.473544 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.473507    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc"
May 25 11:30:53.473967 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.473505    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec88zkfdc" event={"ID":"3bd7bda8-1bcc-4459-bf6c-35623e54139d","Type":"ContainerDied","Data":"0d6f836e3dbc56ae082944e3a83110ac42fbd58874079b2e36aa0d4fb9b3b24b"}
May 25 11:30:53.473967 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.473640    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d6f836e3dbc56ae082944e3a83110ac42fbd58874079b2e36aa0d4fb9b3b24b"
May 25 11:30:53.475318 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.475264    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj"
May 25 11:30:53.475318 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.475280    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b432bfd7b8463c35d1aac5aff9103121dc1483a118ad6caa2a17f39e78jjstj" event={"ID":"a71d7972-772e-40a4-bc14-8c08acc8bd6f","Type":"ContainerDied","Data":"ca04c4d014733791a32e1e0b24fbc87da5dc2df783362a616dfc656c972c56f1"}
May 25 11:30:53.475318 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.475309    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca04c4d014733791a32e1e0b24fbc87da5dc2df783362a616dfc656c972c56f1"
May 25 11:30:53.609202 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.609175    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:53.623193 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.623165    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-bundle\") pod \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") "
May 25 11:30:53.623312 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.623204    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zwdp\" (UniqueName: \"kubernetes.io/projected/ca62d57d-2c20-40bb-997c-8e32a776ecb8-kube-api-access-7zwdp\") pod \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") "
May 25 11:30:53.623312 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.623238    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-util\") pod \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\" (UID: \"ca62d57d-2c20-40bb-997c-8e32a776ecb8\") "
May 25 11:30:53.623644 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.623620    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-bundle" (OuterVolumeSpecName: "bundle") pod "ca62d57d-2c20-40bb-997c-8e32a776ecb8" (UID: "ca62d57d-2c20-40bb-997c-8e32a776ecb8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:53.625715 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.625678    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca62d57d-2c20-40bb-997c-8e32a776ecb8-kube-api-access-7zwdp" (OuterVolumeSpecName: "kube-api-access-7zwdp") pod "ca62d57d-2c20-40bb-997c-8e32a776ecb8" (UID: "ca62d57d-2c20-40bb-997c-8e32a776ecb8"). InnerVolumeSpecName "kube-api-access-7zwdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:30:53.628653 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.628621    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:30:53.629995 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.629973    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-util" (OuterVolumeSpecName: "util") pod "ca62d57d-2c20-40bb-997c-8e32a776ecb8" (UID: "ca62d57d-2c20-40bb-997c-8e32a776ecb8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:53.723866 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.723773    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cww6l\" (UniqueName: \"kubernetes.io/projected/5d230960-8a85-4e7b-9131-d70420b5d609-kube-api-access-cww6l\") pod \"5d230960-8a85-4e7b-9131-d70420b5d609\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") "
May 25 11:30:53.723866 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.723832    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-util\") pod \"5d230960-8a85-4e7b-9131-d70420b5d609\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") "
May 25 11:30:53.723866 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.723861    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-bundle\") pod \"5d230960-8a85-4e7b-9131-d70420b5d609\" (UID: \"5d230960-8a85-4e7b-9131-d70420b5d609\") "
May 25 11:30:53.724154 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.724038    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-util\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:53.724154 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.724056    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca62d57d-2c20-40bb-997c-8e32a776ecb8-bundle\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:53.724154 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.724070    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zwdp\" (UniqueName: \"kubernetes.io/projected/ca62d57d-2c20-40bb-997c-8e32a776ecb8-kube-api-access-7zwdp\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:53.724389 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.724364    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-bundle" (OuterVolumeSpecName: "bundle") pod "5d230960-8a85-4e7b-9131-d70420b5d609" (UID: "5d230960-8a85-4e7b-9131-d70420b5d609"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:53.725955 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.725934    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d230960-8a85-4e7b-9131-d70420b5d609-kube-api-access-cww6l" (OuterVolumeSpecName: "kube-api-access-cww6l") pod "5d230960-8a85-4e7b-9131-d70420b5d609" (UID: "5d230960-8a85-4e7b-9131-d70420b5d609"). InnerVolumeSpecName "kube-api-access-cww6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:30:53.729399 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.729378    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-util" (OuterVolumeSpecName: "util") pod "5d230960-8a85-4e7b-9131-d70420b5d609" (UID: "5d230960-8a85-4e7b-9131-d70420b5d609"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:30:53.825037 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.825002    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cww6l\" (UniqueName: \"kubernetes.io/projected/5d230960-8a85-4e7b-9131-d70420b5d609-kube-api-access-cww6l\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:53.825037 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.825032    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-util\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:53.825037 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:53.825042    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d230960-8a85-4e7b-9131-d70420b5d609-bundle\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:30:54.481076 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:54.481035    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc" event={"ID":"ca62d57d-2c20-40bb-997c-8e32a776ecb8","Type":"ContainerDied","Data":"3d8f06bf241daa822f05cb78e1e84fa455a130752145210910666545959a498f"}
May 25 11:30:54.481076 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:54.481073    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d8f06bf241daa822f05cb78e1e84fa455a130752145210910666545959a498f"
May 25 11:30:54.481076 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:54.481076    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767b92kzc"
May 25 11:30:54.482648 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:54.482624    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk" event={"ID":"5d230960-8a85-4e7b-9131-d70420b5d609","Type":"ContainerDied","Data":"9637b92f10a21ede9c1c66bb42fcb67a66f4728d8c294db0d2fc65612a0e752b"}
May 25 11:30:54.482750 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:54.482655    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9637b92f10a21ede9c1c66bb42fcb67a66f4728d8c294db0d2fc65612a0e752b"
May 25 11:30:54.482750 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:30:54.482673    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30m9xgk"
May 25 11:31:09.251094 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251012    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5"]
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251272    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71d7972-772e-40a4-bc14-8c08acc8bd6f" containerName="extract"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251283    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71d7972-772e-40a4-bc14-8c08acc8bd6f" containerName="extract"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251290    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d230960-8a85-4e7b-9131-d70420b5d609" containerName="extract"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251295    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d230960-8a85-4e7b-9131-d70420b5d609" containerName="extract"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251304    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d230960-8a85-4e7b-9131-d70420b5d609" containerName="util"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251310    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d230960-8a85-4e7b-9131-d70420b5d609" containerName="util"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251317    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62d57d-2c20-40bb-997c-8e32a776ecb8" containerName="pull"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251322    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62d57d-2c20-40bb-997c-8e32a776ecb8" containerName="pull"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251330    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd7bda8-1bcc-4459-bf6c-35623e54139d" containerName="util"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251335    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd7bda8-1bcc-4459-bf6c-35623e54139d" containerName="util"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251343    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71d7972-772e-40a4-bc14-8c08acc8bd6f" containerName="util"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251348    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71d7972-772e-40a4-bc14-8c08acc8bd6f" containerName="util"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251354    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62d57d-2c20-40bb-997c-8e32a776ecb8" containerName="util"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251358    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62d57d-2c20-40bb-997c-8e32a776ecb8" containerName="util"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251368    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd7bda8-1bcc-4459-bf6c-35623e54139d" containerName="pull"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251373    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd7bda8-1bcc-4459-bf6c-35623e54139d" containerName="pull"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251377    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd7bda8-1bcc-4459-bf6c-35623e54139d" containerName="extract"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251382    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd7bda8-1bcc-4459-bf6c-35623e54139d" containerName="extract"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251388    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d230960-8a85-4e7b-9131-d70420b5d609" containerName="pull"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251392    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d230960-8a85-4e7b-9131-d70420b5d609" containerName="pull"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251397    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62d57d-2c20-40bb-997c-8e32a776ecb8" containerName="extract"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251402    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62d57d-2c20-40bb-997c-8e32a776ecb8" containerName="extract"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251412    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71d7972-772e-40a4-bc14-8c08acc8bd6f" containerName="pull"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251417    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71d7972-772e-40a4-bc14-8c08acc8bd6f" containerName="pull"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251480    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="3bd7bda8-1bcc-4459-bf6c-35623e54139d" containerName="extract"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251488    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="ca62d57d-2c20-40bb-997c-8e32a776ecb8" containerName="extract"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251494    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="5d230960-8a85-4e7b-9131-d70420b5d609" containerName="extract"
May 25 11:31:09.251557 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.251500    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a71d7972-772e-40a4-bc14-8c08acc8bd6f" containerName="extract"
May 25 11:31:09.254124 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.254108    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5"
May 25 11:31:09.257036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.257007    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
May 25 11:31:09.257210 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.257049    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5nxc9\""
May 25 11:31:09.257358 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.257344    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 25 11:31:09.258131 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.258113    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 25 11:31:09.279301 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.279266    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5"]
May 25 11:31:09.340665 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.340628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hbjx\" (UniqueName: \"kubernetes.io/projected/909938be-ed0b-4f6d-9819-b8ba2391790f-kube-api-access-4hbjx\") pod \"dns-operator-controller-manager-844548ff4c-nkqx5\" (UID: \"909938be-ed0b-4f6d-9819-b8ba2391790f\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5"
May 25 11:31:09.441767 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.441729    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4hbjx\" (UniqueName: \"kubernetes.io/projected/909938be-ed0b-4f6d-9819-b8ba2391790f-kube-api-access-4hbjx\") pod \"dns-operator-controller-manager-844548ff4c-nkqx5\" (UID: \"909938be-ed0b-4f6d-9819-b8ba2391790f\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5"
May 25 11:31:09.451591 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.451562    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hbjx\" (UniqueName: \"kubernetes.io/projected/909938be-ed0b-4f6d-9819-b8ba2391790f-kube-api-access-4hbjx\") pod \"dns-operator-controller-manager-844548ff4c-nkqx5\" (UID: \"909938be-ed0b-4f6d-9819-b8ba2391790f\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5"
May 25 11:31:09.564352 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.564258    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5"
May 25 11:31:09.696026 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:09.696001    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5"]
May 25 11:31:09.698036 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:31:09.698008    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909938be_ed0b_4f6d_9819_b8ba2391790f.slice/crio-fddbc3ca85799d74f480f5b757f603d0c9a35900b547a2282a1c0f7543ec040d WatchSource:0}: Error finding container fddbc3ca85799d74f480f5b757f603d0c9a35900b547a2282a1c0f7543ec040d: Status 404 returned error can't find the container with id fddbc3ca85799d74f480f5b757f603d0c9a35900b547a2282a1c0f7543ec040d
May 25 11:31:10.544683 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:10.544634    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5" event={"ID":"909938be-ed0b-4f6d-9819-b8ba2391790f","Type":"ContainerStarted","Data":"fddbc3ca85799d74f480f5b757f603d0c9a35900b547a2282a1c0f7543ec040d"}
May 25 11:31:13.556840 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:13.556806    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5" event={"ID":"909938be-ed0b-4f6d-9819-b8ba2391790f","Type":"ContainerStarted","Data":"fe8cb40540e28e9c0cab53e5fea738c4625dc112d1a54921a2235713a7297d64"}
May 25 11:31:13.557210 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:13.556903    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5"
May 25 11:31:13.600001 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:13.599929    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5" podStartSLOduration=0.923098058 podStartE2EDuration="4.599907582s" podCreationTimestamp="2026-05-25 11:31:09 +0000 UTC" firstStartedPulling="2026-05-25 11:31:09.69984894 +0000 UTC m=+529.349133912" lastFinishedPulling="2026-05-25 11:31:13.376658457 +0000 UTC m=+533.025943436" observedRunningTime="2026-05-25 11:31:13.599278058 +0000 UTC m=+533.248563053" watchObservedRunningTime="2026-05-25 11:31:13.599907582 +0000 UTC m=+533.249192578"
May 25 11:31:15.571630 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.571597    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"]
May 25 11:31:15.574842 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.574823    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"
May 25 11:31:15.577660 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.577633    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
May 25 11:31:15.577779 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.577753    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-26pd8\""
May 25 11:31:15.577987 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.577967    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
May 25 11:31:15.585849 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.585824    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"]
May 25 11:31:15.692253 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.692217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb3d9513-2578-471c-a5c5-b18f4a40261d-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-8bs8n\" (UID: \"cb3d9513-2578-471c-a5c5-b18f4a40261d\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"
May 25 11:31:15.692425 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.692260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78scq\" (UniqueName: \"kubernetes.io/projected/cb3d9513-2578-471c-a5c5-b18f4a40261d-kube-api-access-78scq\") pod \"kuadrant-console-plugin-6c886788f8-8bs8n\" (UID: \"cb3d9513-2578-471c-a5c5-b18f4a40261d\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"
May 25 11:31:15.692425 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.692321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb3d9513-2578-471c-a5c5-b18f4a40261d-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-8bs8n\" (UID: \"cb3d9513-2578-471c-a5c5-b18f4a40261d\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"
May 25 11:31:15.792869 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.792826    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb3d9513-2578-471c-a5c5-b18f4a40261d-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-8bs8n\" (UID: \"cb3d9513-2578-471c-a5c5-b18f4a40261d\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"
May 25 11:31:15.792869 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.792874    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-78scq\" (UniqueName: \"kubernetes.io/projected/cb3d9513-2578-471c-a5c5-b18f4a40261d-kube-api-access-78scq\") pod \"kuadrant-console-plugin-6c886788f8-8bs8n\" (UID: \"cb3d9513-2578-471c-a5c5-b18f4a40261d\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"
May 25 11:31:15.793076 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.792997    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb3d9513-2578-471c-a5c5-b18f4a40261d-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-8bs8n\" (UID: \"cb3d9513-2578-471c-a5c5-b18f4a40261d\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"
May 25 11:31:15.793558 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.793528    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb3d9513-2578-471c-a5c5-b18f4a40261d-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-8bs8n\" (UID: \"cb3d9513-2578-471c-a5c5-b18f4a40261d\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"
May 25 11:31:15.795544 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.795522    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb3d9513-2578-471c-a5c5-b18f4a40261d-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-8bs8n\" (UID: \"cb3d9513-2578-471c-a5c5-b18f4a40261d\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"
May 25 11:31:15.803120 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.803094    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78scq\" (UniqueName: \"kubernetes.io/projected/cb3d9513-2578-471c-a5c5-b18f4a40261d-kube-api-access-78scq\") pod \"kuadrant-console-plugin-6c886788f8-8bs8n\" (UID: \"cb3d9513-2578-471c-a5c5-b18f4a40261d\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"
May 25 11:31:15.884098 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:15.884009    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"
May 25 11:31:16.027908 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:16.027823    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n"]
May 25 11:31:16.030016 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:31:16.029984    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb3d9513_2578_471c_a5c5_b18f4a40261d.slice/crio-196a79807bcd72509c30634d2c157d7bd4473d1ea533cbd5482d5cfdfd0004fb WatchSource:0}: Error finding container 196a79807bcd72509c30634d2c157d7bd4473d1ea533cbd5482d5cfdfd0004fb: Status 404 returned error can't find the container with id 196a79807bcd72509c30634d2c157d7bd4473d1ea533cbd5482d5cfdfd0004fb
May 25 11:31:16.572818 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:16.572780    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n" event={"ID":"cb3d9513-2578-471c-a5c5-b18f4a40261d","Type":"ContainerStarted","Data":"196a79807bcd72509c30634d2c157d7bd4473d1ea533cbd5482d5cfdfd0004fb"}
May 25 11:31:21.592530 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:21.592488    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n" event={"ID":"cb3d9513-2578-471c-a5c5-b18f4a40261d","Type":"ContainerStarted","Data":"5798e648bb9404c6f51f7a5dbabadb907926334ee41838c6d0aabf0248d8f18d"}
May 25 11:31:21.616648 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:21.616595    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-8bs8n" podStartSLOduration=2.080586108 podStartE2EDuration="6.616580636s" podCreationTimestamp="2026-05-25 11:31:15 +0000 UTC" firstStartedPulling="2026-05-25 11:31:16.031427046 +0000 UTC m=+535.680712023" lastFinishedPulling="2026-05-25 11:31:20.567421579 +0000 UTC m=+540.216706551" observedRunningTime="2026-05-25 11:31:21.615970926 +0000 UTC m=+541.265255924" watchObservedRunningTime="2026-05-25 11:31:21.616580636 +0000 UTC m=+541.265865630"
May 25 11:31:24.563226 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:24.563192    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-nkqx5"
May 25 11:31:58.965075 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:58.965038    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-674b59b84c-r724p"]
May 25 11:31:58.988857 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:58.988829    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-r724p"]
May 25 11:31:58.989020 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:58.988941    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-r724p"
May 25 11:31:58.991808 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:58.991778    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7dqhj\""
May 25 11:31:59.070571 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.070540    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79cbc94b89-2crwt"]
May 25 11:31:59.074838 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.074807    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-2crwt"
May 25 11:31:59.080958 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.080929    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-2crwt"]
May 25 11:31:59.139394 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.139358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpt6n\" (UniqueName: \"kubernetes.io/projected/4d5c7345-849f-44f6-82c9-49b9077e656b-kube-api-access-jpt6n\") pod \"authorino-674b59b84c-r724p\" (UID: \"4d5c7345-849f-44f6-82c9-49b9077e656b\") " pod="kuadrant-system/authorino-674b59b84c-r724p"
May 25 11:31:59.240720 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.240685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5kzz\" (UniqueName: \"kubernetes.io/projected/b2ed330d-0e41-4175-aa4a-250def6e4944-kube-api-access-l5kzz\") pod \"authorino-79cbc94b89-2crwt\" (UID: \"b2ed330d-0e41-4175-aa4a-250def6e4944\") " pod="kuadrant-system/authorino-79cbc94b89-2crwt"
May 25 11:31:59.240921 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.240732    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jpt6n\" (UniqueName: \"kubernetes.io/projected/4d5c7345-849f-44f6-82c9-49b9077e656b-kube-api-access-jpt6n\") pod \"authorino-674b59b84c-r724p\" (UID: \"4d5c7345-849f-44f6-82c9-49b9077e656b\") " pod="kuadrant-system/authorino-674b59b84c-r724p"
May 25 11:31:59.252676 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.252641    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpt6n\" (UniqueName: \"kubernetes.io/projected/4d5c7345-849f-44f6-82c9-49b9077e656b-kube-api-access-jpt6n\") pod \"authorino-674b59b84c-r724p\" (UID: \"4d5c7345-849f-44f6-82c9-49b9077e656b\") " pod="kuadrant-system/authorino-674b59b84c-r724p"
May 25 11:31:59.297775 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.297720    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-r724p"
May 25 11:31:59.342676 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.341985    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l5kzz\" (UniqueName: \"kubernetes.io/projected/b2ed330d-0e41-4175-aa4a-250def6e4944-kube-api-access-l5kzz\") pod \"authorino-79cbc94b89-2crwt\" (UID: \"b2ed330d-0e41-4175-aa4a-250def6e4944\") " pod="kuadrant-system/authorino-79cbc94b89-2crwt"
May 25 11:31:59.351356 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.351317    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5kzz\" (UniqueName: \"kubernetes.io/projected/b2ed330d-0e41-4175-aa4a-250def6e4944-kube-api-access-l5kzz\") pod \"authorino-79cbc94b89-2crwt\" (UID: \"b2ed330d-0e41-4175-aa4a-250def6e4944\") " pod="kuadrant-system/authorino-79cbc94b89-2crwt"
May 25 11:31:59.385639 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.385608    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-2crwt"
May 25 11:31:59.432823 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.432794    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-r724p"]
May 25 11:31:59.434387 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:31:59.434361    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d5c7345_849f_44f6_82c9_49b9077e656b.slice/crio-55d2c123c3d404a5890e5933c1345290fc1b875d257feefc10be7082c4cc57d9 WatchSource:0}: Error finding container 55d2c123c3d404a5890e5933c1345290fc1b875d257feefc10be7082c4cc57d9: Status 404 returned error can't find the container with id 55d2c123c3d404a5890e5933c1345290fc1b875d257feefc10be7082c4cc57d9
May 25 11:31:59.523276 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.523253    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-2crwt"]
May 25 11:31:59.524727 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:31:59.524699    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2ed330d_0e41_4175_aa4a_250def6e4944.slice/crio-1e12276b925061aae00da50dec1f6d9bac0c25357300bb59753da3add8ae8d9f WatchSource:0}: Error finding container 1e12276b925061aae00da50dec1f6d9bac0c25357300bb59753da3add8ae8d9f: Status 404 returned error can't find the container with id 1e12276b925061aae00da50dec1f6d9bac0c25357300bb59753da3add8ae8d9f
May 25 11:31:59.727391 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.727352    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-r724p" event={"ID":"4d5c7345-849f-44f6-82c9-49b9077e656b","Type":"ContainerStarted","Data":"55d2c123c3d404a5890e5933c1345290fc1b875d257feefc10be7082c4cc57d9"}
May 25 11:31:59.728308 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:31:59.728280    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-2crwt" event={"ID":"b2ed330d-0e41-4175-aa4a-250def6e4944","Type":"ContainerStarted","Data":"1e12276b925061aae00da50dec1f6d9bac0c25357300bb59753da3add8ae8d9f"}
May 25 11:32:02.743371 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:02.743333    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-r724p" event={"ID":"4d5c7345-849f-44f6-82c9-49b9077e656b","Type":"ContainerStarted","Data":"f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce"}
May 25 11:32:02.744677 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:02.744651    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-2crwt" event={"ID":"b2ed330d-0e41-4175-aa4a-250def6e4944","Type":"ContainerStarted","Data":"97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f"}
May 25 11:32:02.784831 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:02.784783    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-674b59b84c-r724p" podStartSLOduration=2.110012404 podStartE2EDuration="4.784768978s" podCreationTimestamp="2026-05-25 11:31:58 +0000 UTC" firstStartedPulling="2026-05-25 11:31:59.435862973 +0000 UTC m=+579.085147945" lastFinishedPulling="2026-05-25 11:32:02.110619547 +0000 UTC m=+581.759904519" observedRunningTime="2026-05-25 11:32:02.761056614 +0000 UTC m=+582.410341609" watchObservedRunningTime="2026-05-25 11:32:02.784768978 +0000 UTC m=+582.434054039"
May 25 11:32:02.786127 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:02.786090    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-79cbc94b89-2crwt" podStartSLOduration=1.212377597 podStartE2EDuration="3.786080854s" podCreationTimestamp="2026-05-25 11:31:59 +0000 UTC" firstStartedPulling="2026-05-25 11:31:59.52604113 +0000 UTC m=+579.175326105" lastFinishedPulling="2026-05-25 11:32:02.099744383 +0000 UTC m=+581.749029362" observedRunningTime="2026-05-25 11:32:02.784395861 +0000 UTC m=+582.433680856" watchObservedRunningTime="2026-05-25 11:32:02.786080854 +0000 UTC m=+582.435365848"
May 25 11:32:02.810832 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:02.810800    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-r724p"]
May 25 11:32:04.752124 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:04.752079    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-674b59b84c-r724p" podUID="4d5c7345-849f-44f6-82c9-49b9077e656b" containerName="authorino" containerID="cri-o://f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce" gracePeriod=30
May 25 11:32:04.994164 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:04.994140    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-r724p"
May 25 11:32:05.087898 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.087801    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpt6n\" (UniqueName: \"kubernetes.io/projected/4d5c7345-849f-44f6-82c9-49b9077e656b-kube-api-access-jpt6n\") pod \"4d5c7345-849f-44f6-82c9-49b9077e656b\" (UID: \"4d5c7345-849f-44f6-82c9-49b9077e656b\") "
May 25 11:32:05.090184 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.090154    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5c7345-849f-44f6-82c9-49b9077e656b-kube-api-access-jpt6n" (OuterVolumeSpecName: "kube-api-access-jpt6n") pod "4d5c7345-849f-44f6-82c9-49b9077e656b" (UID: "4d5c7345-849f-44f6-82c9-49b9077e656b"). InnerVolumeSpecName "kube-api-access-jpt6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:32:05.189024 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.188988    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpt6n\" (UniqueName: \"kubernetes.io/projected/4d5c7345-849f-44f6-82c9-49b9077e656b-kube-api-access-jpt6n\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:32:05.756295 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.756261    2578 generic.go:358] "Generic (PLEG): container finished" podID="4d5c7345-849f-44f6-82c9-49b9077e656b" containerID="f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce" exitCode=0
May 25 11:32:05.756790 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.756315    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-r724p"
May 25 11:32:05.756790 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.756330    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-r724p" event={"ID":"4d5c7345-849f-44f6-82c9-49b9077e656b","Type":"ContainerDied","Data":"f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce"}
May 25 11:32:05.756790 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.756357    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-r724p" event={"ID":"4d5c7345-849f-44f6-82c9-49b9077e656b","Type":"ContainerDied","Data":"55d2c123c3d404a5890e5933c1345290fc1b875d257feefc10be7082c4cc57d9"}
May 25 11:32:05.756790 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.756372    2578 scope.go:117] "RemoveContainer" containerID="f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce"
May 25 11:32:05.766681 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.766635    2578 scope.go:117] "RemoveContainer" containerID="f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce"
May 25 11:32:05.767100 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:32:05.767079    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce\": container with ID starting with f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce not found: ID does not exist" containerID="f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce"
May 25 11:32:05.767164 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.767110    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce"} err="failed to get container status \"f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce\": rpc error: code = NotFound desc = could not find container \"f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce\": container with ID starting with f8165eb0219c4732f26005eaace4e4f9b9cc22362f354f998cfae512851f73ce not found: ID does not exist"
May 25 11:32:05.777393 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.777363    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-r724p"]
May 25 11:32:05.783651 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:05.783627    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-674b59b84c-r724p"]
May 25 11:32:06.934775 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:06.934739    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d5c7345-849f-44f6-82c9-49b9077e656b" path="/var/lib/kubelet/pods/4d5c7345-849f-44f6-82c9-49b9077e656b/volumes"
May 25 11:32:20.888512 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:20.888478    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:32:20.888940 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:20.888476    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:32:23.828404 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:23.828369    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68bd676465-mgvhp"]
May 25 11:32:23.828789 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:23.828646    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5c7345-849f-44f6-82c9-49b9077e656b" containerName="authorino"
May 25 11:32:23.828789 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:23.828662    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5c7345-849f-44f6-82c9-49b9077e656b" containerName="authorino"
May 25 11:32:23.828789 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:23.828759    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="4d5c7345-849f-44f6-82c9-49b9077e656b" containerName="authorino"
May 25 11:32:23.832675 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:23.832656    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-mgvhp"
May 25 11:32:23.836624 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:23.836590    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
May 25 11:32:23.845896 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:23.845872    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-mgvhp"]
May 25 11:32:23.927287 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:23.927253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff29n\" (UniqueName: \"kubernetes.io/projected/39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d-kube-api-access-ff29n\") pod \"authorino-68bd676465-mgvhp\" (UID: \"39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d\") " pod="kuadrant-system/authorino-68bd676465-mgvhp"
May 25 11:32:23.927287 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:23.927286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d-tls-cert\") pod \"authorino-68bd676465-mgvhp\" (UID: \"39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d\") " pod="kuadrant-system/authorino-68bd676465-mgvhp"
May 25 11:32:24.028253 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:24.028220    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ff29n\" (UniqueName: \"kubernetes.io/projected/39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d-kube-api-access-ff29n\") pod \"authorino-68bd676465-mgvhp\" (UID: \"39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d\") " pod="kuadrant-system/authorino-68bd676465-mgvhp"
May 25 11:32:24.028253 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:24.028257    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d-tls-cert\") pod \"authorino-68bd676465-mgvhp\" (UID: \"39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d\") " pod="kuadrant-system/authorino-68bd676465-mgvhp"
May 25 11:32:24.030957 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:24.030938    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d-tls-cert\") pod \"authorino-68bd676465-mgvhp\" (UID: \"39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d\") " pod="kuadrant-system/authorino-68bd676465-mgvhp"
May 25 11:32:24.038948 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:24.038927    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff29n\" (UniqueName: \"kubernetes.io/projected/39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d-kube-api-access-ff29n\") pod \"authorino-68bd676465-mgvhp\" (UID: \"39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d\") " pod="kuadrant-system/authorino-68bd676465-mgvhp"
May 25 11:32:24.141876 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:24.141787    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-mgvhp"
May 25 11:32:24.272745 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:24.272722    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-mgvhp"]
May 25 11:32:24.274325 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:32:24.274294    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39cfa8b7_72d6_44fe_8a85_5f4e05d7d26d.slice/crio-277332f7f34bd2850f5ef1348350867b30d19241a90d85e6f9890326b609e10d WatchSource:0}: Error finding container 277332f7f34bd2850f5ef1348350867b30d19241a90d85e6f9890326b609e10d: Status 404 returned error can't find the container with id 277332f7f34bd2850f5ef1348350867b30d19241a90d85e6f9890326b609e10d
May 25 11:32:24.824604 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:24.824565    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-mgvhp" event={"ID":"39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d","Type":"ContainerStarted","Data":"277332f7f34bd2850f5ef1348350867b30d19241a90d85e6f9890326b609e10d"}
May 25 11:32:25.829671 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:25.829631    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-mgvhp" event={"ID":"39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d","Type":"ContainerStarted","Data":"64b4029fe367cd70a1b84e0e2102c25dc0716ee50bfbfe49fd7d61cf5cd522e9"}
May 25 11:32:25.847290 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:25.847238    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-68bd676465-mgvhp" podStartSLOduration=2.243370208 podStartE2EDuration="2.84722301s" podCreationTimestamp="2026-05-25 11:32:23 +0000 UTC" firstStartedPulling="2026-05-25 11:32:24.275702872 +0000 UTC m=+603.924987848" lastFinishedPulling="2026-05-25 11:32:24.879555676 +0000 UTC m=+604.528840650" observedRunningTime="2026-05-25 11:32:25.845581354 +0000 UTC m=+605.494866352" watchObservedRunningTime="2026-05-25 11:32:25.84722301 +0000 UTC m=+605.496508038"
May 25 11:32:25.874655 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:25.874619    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-2crwt"]
May 25 11:32:25.874927 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:25.874885    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-79cbc94b89-2crwt" podUID="b2ed330d-0e41-4175-aa4a-250def6e4944" containerName="authorino" containerID="cri-o://97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f" gracePeriod=30
May 25 11:32:26.122647 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.122626    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-2crwt"
May 25 11:32:26.246915 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.246880    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5kzz\" (UniqueName: \"kubernetes.io/projected/b2ed330d-0e41-4175-aa4a-250def6e4944-kube-api-access-l5kzz\") pod \"b2ed330d-0e41-4175-aa4a-250def6e4944\" (UID: \"b2ed330d-0e41-4175-aa4a-250def6e4944\") "
May 25 11:32:26.249082 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.249056    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ed330d-0e41-4175-aa4a-250def6e4944-kube-api-access-l5kzz" (OuterVolumeSpecName: "kube-api-access-l5kzz") pod "b2ed330d-0e41-4175-aa4a-250def6e4944" (UID: "b2ed330d-0e41-4175-aa4a-250def6e4944"). InnerVolumeSpecName "kube-api-access-l5kzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:32:26.347575 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.347524    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5kzz\" (UniqueName: \"kubernetes.io/projected/b2ed330d-0e41-4175-aa4a-250def6e4944-kube-api-access-l5kzz\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:32:26.833895 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.833854    2578 generic.go:358] "Generic (PLEG): container finished" podID="b2ed330d-0e41-4175-aa4a-250def6e4944" containerID="97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f" exitCode=0
May 25 11:32:26.834324 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.833906    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-2crwt"
May 25 11:32:26.834324 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.833943    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-2crwt" event={"ID":"b2ed330d-0e41-4175-aa4a-250def6e4944","Type":"ContainerDied","Data":"97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f"}
May 25 11:32:26.834324 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.833977    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-2crwt" event={"ID":"b2ed330d-0e41-4175-aa4a-250def6e4944","Type":"ContainerDied","Data":"1e12276b925061aae00da50dec1f6d9bac0c25357300bb59753da3add8ae8d9f"}
May 25 11:32:26.834324 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.833993    2578 scope.go:117] "RemoveContainer" containerID="97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f"
May 25 11:32:26.842596 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.842580    2578 scope.go:117] "RemoveContainer" containerID="97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f"
May 25 11:32:26.842855 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:32:26.842836    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f\": container with ID starting with 97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f not found: ID does not exist" containerID="97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f"
May 25 11:32:26.842897 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.842863    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f"} err="failed to get container status \"97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f\": rpc error: code = NotFound desc = could not find container \"97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f\": container with ID starting with 97771107ace895ab4ea8f84e8c5117f518d3b8af24cc0b8fda65354a13b73e7f not found: ID does not exist"
May 25 11:32:26.855546 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.855515    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-2crwt"]
May 25 11:32:26.862631 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.862608    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-2crwt"]
May 25 11:32:26.939108 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:26.935918    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2ed330d-0e41-4175-aa4a-250def6e4944" path="/var/lib/kubelet/pods/b2ed330d-0e41-4175-aa4a-250def6e4944/volumes"
May 25 11:32:45.553401 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.553317    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-dbwql"]
May 25 11:32:45.555806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.553597    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ed330d-0e41-4175-aa4a-250def6e4944" containerName="authorino"
May 25 11:32:45.555806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.553607    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ed330d-0e41-4175-aa4a-250def6e4944" containerName="authorino"
May 25 11:32:45.555806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.553664    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="b2ed330d-0e41-4175-aa4a-250def6e4944" containerName="authorino"
May 25 11:32:45.556689 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.556673    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-dbwql"
May 25 11:32:45.559555 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.559522    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 11:32:45.559555 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.559525    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 25 11:32:45.559757 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.559549    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 11:32:45.560683 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.560666    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-nbhs4\""
May 25 11:32:45.566191 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.566164    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-dbwql"]
May 25 11:32:45.710808 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.710776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12dfe165-4edb-48a2-8b52-34d653fa7989-data\") pod \"seaweedfs-79f457656b-dbwql\" (UID: \"12dfe165-4edb-48a2-8b52-34d653fa7989\") " pod="kserve/seaweedfs-79f457656b-dbwql"
May 25 11:32:45.710994 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.710830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tdd\" (UniqueName: \"kubernetes.io/projected/12dfe165-4edb-48a2-8b52-34d653fa7989-kube-api-access-28tdd\") pod \"seaweedfs-79f457656b-dbwql\" (UID: \"12dfe165-4edb-48a2-8b52-34d653fa7989\") " pod="kserve/seaweedfs-79f457656b-dbwql"
May 25 11:32:45.811994 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.811899    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12dfe165-4edb-48a2-8b52-34d653fa7989-data\") pod \"seaweedfs-79f457656b-dbwql\" (UID: \"12dfe165-4edb-48a2-8b52-34d653fa7989\") " pod="kserve/seaweedfs-79f457656b-dbwql"
May 25 11:32:45.811994 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.811970    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-28tdd\" (UniqueName: \"kubernetes.io/projected/12dfe165-4edb-48a2-8b52-34d653fa7989-kube-api-access-28tdd\") pod \"seaweedfs-79f457656b-dbwql\" (UID: \"12dfe165-4edb-48a2-8b52-34d653fa7989\") " pod="kserve/seaweedfs-79f457656b-dbwql"
May 25 11:32:45.812306 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.812279    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12dfe165-4edb-48a2-8b52-34d653fa7989-data\") pod \"seaweedfs-79f457656b-dbwql\" (UID: \"12dfe165-4edb-48a2-8b52-34d653fa7989\") " pod="kserve/seaweedfs-79f457656b-dbwql"
May 25 11:32:45.821041 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.821009    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tdd\" (UniqueName: \"kubernetes.io/projected/12dfe165-4edb-48a2-8b52-34d653fa7989-kube-api-access-28tdd\") pod \"seaweedfs-79f457656b-dbwql\" (UID: \"12dfe165-4edb-48a2-8b52-34d653fa7989\") " pod="kserve/seaweedfs-79f457656b-dbwql"
May 25 11:32:45.867035 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:45.866995    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-dbwql"
May 25 11:32:46.014881 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:46.014849    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-dbwql"]
May 25 11:32:46.018363 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:32:46.018336    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12dfe165_4edb_48a2_8b52_34d653fa7989.slice/crio-afd8f6732fcd2ed0a884897a37b08ec9be12e7edc2f543268b7a3f548aa582f6 WatchSource:0}: Error finding container afd8f6732fcd2ed0a884897a37b08ec9be12e7edc2f543268b7a3f548aa582f6: Status 404 returned error can't find the container with id afd8f6732fcd2ed0a884897a37b08ec9be12e7edc2f543268b7a3f548aa582f6
May 25 11:32:46.019897 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:46.019882    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:32:46.905822 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:46.905783    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-dbwql" event={"ID":"12dfe165-4edb-48a2-8b52-34d653fa7989","Type":"ContainerStarted","Data":"afd8f6732fcd2ed0a884897a37b08ec9be12e7edc2f543268b7a3f548aa582f6"}
May 25 11:32:49.920199 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:49.920107    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-dbwql" event={"ID":"12dfe165-4edb-48a2-8b52-34d653fa7989","Type":"ContainerStarted","Data":"f3cf2c0d368b37bf690d453ce30c04832e810cbb49dd05b32e9136efe6abb6d7"}
May 25 11:32:49.920199 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:49.920173    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-dbwql"
May 25 11:32:49.939673 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:49.939616    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-dbwql" podStartSLOduration=1.479583464 podStartE2EDuration="4.939602646s" podCreationTimestamp="2026-05-25 11:32:45 +0000 UTC" firstStartedPulling="2026-05-25 11:32:46.01999817 +0000 UTC m=+625.669283143" lastFinishedPulling="2026-05-25 11:32:49.480017349 +0000 UTC m=+629.129302325" observedRunningTime="2026-05-25 11:32:49.937733878 +0000 UTC m=+629.587018872" watchObservedRunningTime="2026-05-25 11:32:49.939602646 +0000 UTC m=+629.588887640"
May 25 11:32:55.925495 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:32:55.925435    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-dbwql"
May 25 11:33:56.869372 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.869340    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-2sqrq"]
May 25 11:33:56.872719 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.872699    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-2sqrq"
May 25 11:33:56.875518 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.875499    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-kc2pq\""
May 25 11:33:56.875623 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.875578    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 25 11:33:56.881189 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.881165    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-2sqrq"]
May 25 11:33:56.898610 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.898580    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-8xqts"]
May 25 11:33:56.901707 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.901684    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-8xqts"
May 25 11:33:56.904415 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.904393    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 25 11:33:56.904534 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.904400    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-bg6xm\""
May 25 11:33:56.910066 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.910042    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-8xqts"]
May 25 11:33:56.975174 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.975139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5jkn\" (UniqueName: \"kubernetes.io/projected/9a3e73f0-be2e-4c2d-9688-0ef36119fe6d-kube-api-access-p5jkn\") pod \"odh-model-controller-696fc77849-8xqts\" (UID: \"9a3e73f0-be2e-4c2d-9688-0ef36119fe6d\") " pod="kserve/odh-model-controller-696fc77849-8xqts"
May 25 11:33:56.975357 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.975183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96002196-66a8-4257-aa19-6ac0b2bf6a05-tls-certs\") pod \"model-serving-api-86f7b4b499-2sqrq\" (UID: \"96002196-66a8-4257-aa19-6ac0b2bf6a05\") " pod="kserve/model-serving-api-86f7b4b499-2sqrq"
May 25 11:33:56.975357 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.975235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a3e73f0-be2e-4c2d-9688-0ef36119fe6d-cert\") pod \"odh-model-controller-696fc77849-8xqts\" (UID: \"9a3e73f0-be2e-4c2d-9688-0ef36119fe6d\") " pod="kserve/odh-model-controller-696fc77849-8xqts"
May 25 11:33:56.975357 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:56.975275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxfw\" (UniqueName: \"kubernetes.io/projected/96002196-66a8-4257-aa19-6ac0b2bf6a05-kube-api-access-8bxfw\") pod \"model-serving-api-86f7b4b499-2sqrq\" (UID: \"96002196-66a8-4257-aa19-6ac0b2bf6a05\") " pod="kserve/model-serving-api-86f7b4b499-2sqrq"
May 25 11:33:57.076114 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.076073    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96002196-66a8-4257-aa19-6ac0b2bf6a05-tls-certs\") pod \"model-serving-api-86f7b4b499-2sqrq\" (UID: \"96002196-66a8-4257-aa19-6ac0b2bf6a05\") " pod="kserve/model-serving-api-86f7b4b499-2sqrq"
May 25 11:33:57.076114 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.076120    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a3e73f0-be2e-4c2d-9688-0ef36119fe6d-cert\") pod \"odh-model-controller-696fc77849-8xqts\" (UID: \"9a3e73f0-be2e-4c2d-9688-0ef36119fe6d\") " pod="kserve/odh-model-controller-696fc77849-8xqts"
May 25 11:33:57.076337 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.076150    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxfw\" (UniqueName: \"kubernetes.io/projected/96002196-66a8-4257-aa19-6ac0b2bf6a05-kube-api-access-8bxfw\") pod \"model-serving-api-86f7b4b499-2sqrq\" (UID: \"96002196-66a8-4257-aa19-6ac0b2bf6a05\") " pod="kserve/model-serving-api-86f7b4b499-2sqrq"
May 25 11:33:57.076337 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.076191    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jkn\" (UniqueName: \"kubernetes.io/projected/9a3e73f0-be2e-4c2d-9688-0ef36119fe6d-kube-api-access-p5jkn\") pod \"odh-model-controller-696fc77849-8xqts\" (UID: \"9a3e73f0-be2e-4c2d-9688-0ef36119fe6d\") " pod="kserve/odh-model-controller-696fc77849-8xqts"
May 25 11:33:57.078806 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.078780    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a3e73f0-be2e-4c2d-9688-0ef36119fe6d-cert\") pod \"odh-model-controller-696fc77849-8xqts\" (UID: \"9a3e73f0-be2e-4c2d-9688-0ef36119fe6d\") " pod="kserve/odh-model-controller-696fc77849-8xqts"
May 25 11:33:57.078928 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.078782    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96002196-66a8-4257-aa19-6ac0b2bf6a05-tls-certs\") pod \"model-serving-api-86f7b4b499-2sqrq\" (UID: \"96002196-66a8-4257-aa19-6ac0b2bf6a05\") " pod="kserve/model-serving-api-86f7b4b499-2sqrq"
May 25 11:33:57.089080 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.089048    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxfw\" (UniqueName: \"kubernetes.io/projected/96002196-66a8-4257-aa19-6ac0b2bf6a05-kube-api-access-8bxfw\") pod \"model-serving-api-86f7b4b499-2sqrq\" (UID: \"96002196-66a8-4257-aa19-6ac0b2bf6a05\") " pod="kserve/model-serving-api-86f7b4b499-2sqrq"
May 25 11:33:57.089303 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.089283    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5jkn\" (UniqueName: \"kubernetes.io/projected/9a3e73f0-be2e-4c2d-9688-0ef36119fe6d-kube-api-access-p5jkn\") pod \"odh-model-controller-696fc77849-8xqts\" (UID: \"9a3e73f0-be2e-4c2d-9688-0ef36119fe6d\") " pod="kserve/odh-model-controller-696fc77849-8xqts"
May 25 11:33:57.183272 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.183170    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-2sqrq"
May 25 11:33:57.212039 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.212010    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-8xqts"
May 25 11:33:57.331449 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.331404    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-2sqrq"]
May 25 11:33:57.332941 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:33:57.332913    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96002196_66a8_4257_aa19_6ac0b2bf6a05.slice/crio-aaccb4fe51a984fdd68eb87f3545bdade8e7dc251ef69c0a81aa0f4dee1088af WatchSource:0}: Error finding container aaccb4fe51a984fdd68eb87f3545bdade8e7dc251ef69c0a81aa0f4dee1088af: Status 404 returned error can't find the container with id aaccb4fe51a984fdd68eb87f3545bdade8e7dc251ef69c0a81aa0f4dee1088af
May 25 11:33:57.351869 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:57.351843    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-8xqts"]
May 25 11:33:57.353672 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:33:57.353646    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3e73f0_be2e_4c2d_9688_0ef36119fe6d.slice/crio-8ea8cd62f51ce28b9f4da1094520e2c0f411901b41f01da0a7c09b6ade1ef774 WatchSource:0}: Error finding container 8ea8cd62f51ce28b9f4da1094520e2c0f411901b41f01da0a7c09b6ade1ef774: Status 404 returned error can't find the container with id 8ea8cd62f51ce28b9f4da1094520e2c0f411901b41f01da0a7c09b6ade1ef774
May 25 11:33:58.154605 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:58.154556    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-8xqts" event={"ID":"9a3e73f0-be2e-4c2d-9688-0ef36119fe6d","Type":"ContainerStarted","Data":"8ea8cd62f51ce28b9f4da1094520e2c0f411901b41f01da0a7c09b6ade1ef774"}
May 25 11:33:58.157602 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:33:58.157574    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-2sqrq" event={"ID":"96002196-66a8-4257-aa19-6ac0b2bf6a05","Type":"ContainerStarted","Data":"aaccb4fe51a984fdd68eb87f3545bdade8e7dc251ef69c0a81aa0f4dee1088af"}
May 25 11:34:02.177925 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:02.177887    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-8xqts" event={"ID":"9a3e73f0-be2e-4c2d-9688-0ef36119fe6d","Type":"ContainerStarted","Data":"bd781a5ac61113d832e7f9e1a6fe98b5818da37d5bf4b02fb1c57c5a4ab34452"}
May 25 11:34:02.178404 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:02.177992    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-8xqts"
May 25 11:34:02.179386 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:02.179363    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-2sqrq" event={"ID":"96002196-66a8-4257-aa19-6ac0b2bf6a05","Type":"ContainerStarted","Data":"c86532f80e05f05886f3c60e240a0654c7b68a10db6d51f58d549bb2c3504ff1"}
May 25 11:34:02.179514 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:02.179493    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-2sqrq"
May 25 11:34:02.216621 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:02.216567    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-8xqts" podStartSLOduration=2.421203188 podStartE2EDuration="6.216553129s" podCreationTimestamp="2026-05-25 11:33:56 +0000 UTC" firstStartedPulling="2026-05-25 11:33:57.354933815 +0000 UTC m=+697.004218790" lastFinishedPulling="2026-05-25 11:34:01.150283755 +0000 UTC m=+700.799568731" observedRunningTime="2026-05-25 11:34:02.214900755 +0000 UTC m=+701.864185750" watchObservedRunningTime="2026-05-25 11:34:02.216553129 +0000 UTC m=+701.865838123"
May 25 11:34:02.260648 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:02.260596    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-2sqrq" podStartSLOduration=2.38559753 podStartE2EDuration="6.260580045s" podCreationTimestamp="2026-05-25 11:33:56 +0000 UTC" firstStartedPulling="2026-05-25 11:33:57.334895222 +0000 UTC m=+696.984180196" lastFinishedPulling="2026-05-25 11:34:01.20987773 +0000 UTC m=+700.859162711" observedRunningTime="2026-05-25 11:34:02.25959447 +0000 UTC m=+701.908879475" watchObservedRunningTime="2026-05-25 11:34:02.260580045 +0000 UTC m=+701.909865039"
May 25 11:34:13.184820 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:13.184735    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-8xqts"
May 25 11:34:13.186663 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:13.186642    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-2sqrq"
May 25 11:34:14.046018 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:14.045980    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-h56gh"]
May 25 11:34:14.049178 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:14.049158    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-h56gh"
May 25 11:34:14.060847 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:14.060822    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-h56gh"]
May 25 11:34:14.227206 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:14.227169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knsdh\" (UniqueName: \"kubernetes.io/projected/241b75d1-a355-4b2b-84d2-4bf4e81ad0ac-kube-api-access-knsdh\") pod \"s3-init-h56gh\" (UID: \"241b75d1-a355-4b2b-84d2-4bf4e81ad0ac\") " pod="kserve/s3-init-h56gh"
May 25 11:34:14.328036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:14.327929    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-knsdh\" (UniqueName: \"kubernetes.io/projected/241b75d1-a355-4b2b-84d2-4bf4e81ad0ac-kube-api-access-knsdh\") pod \"s3-init-h56gh\" (UID: \"241b75d1-a355-4b2b-84d2-4bf4e81ad0ac\") " pod="kserve/s3-init-h56gh"
May 25 11:34:14.337443 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:14.337418    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knsdh\" (UniqueName: \"kubernetes.io/projected/241b75d1-a355-4b2b-84d2-4bf4e81ad0ac-kube-api-access-knsdh\") pod \"s3-init-h56gh\" (UID: \"241b75d1-a355-4b2b-84d2-4bf4e81ad0ac\") " pod="kserve/s3-init-h56gh"
May 25 11:34:14.358362 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:14.358337    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-h56gh"
May 25 11:34:14.489044 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:14.489020    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-h56gh"]
May 25 11:34:14.490853 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:34:14.490811    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod241b75d1_a355_4b2b_84d2_4bf4e81ad0ac.slice/crio-eb5e812caca92ae0761ea5d4c4252598c62f6f1e52fed60958300447c29f69ab WatchSource:0}: Error finding container eb5e812caca92ae0761ea5d4c4252598c62f6f1e52fed60958300447c29f69ab: Status 404 returned error can't find the container with id eb5e812caca92ae0761ea5d4c4252598c62f6f1e52fed60958300447c29f69ab
May 25 11:34:15.227146 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:15.227103    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-h56gh" event={"ID":"241b75d1-a355-4b2b-84d2-4bf4e81ad0ac","Type":"ContainerStarted","Data":"eb5e812caca92ae0761ea5d4c4252598c62f6f1e52fed60958300447c29f69ab"}
May 25 11:34:19.245810 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:19.245754    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-h56gh" event={"ID":"241b75d1-a355-4b2b-84d2-4bf4e81ad0ac","Type":"ContainerStarted","Data":"15e09614b0ecff665d69f376a2650284b5ea403c09c4194154fa097381269730"}
May 25 11:34:19.265646 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:19.265598    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-h56gh" podStartSLOduration=0.87630149 podStartE2EDuration="5.265583244s" podCreationTimestamp="2026-05-25 11:34:14 +0000 UTC" firstStartedPulling="2026-05-25 11:34:14.492694817 +0000 UTC m=+714.141979790" lastFinishedPulling="2026-05-25 11:34:18.881976572 +0000 UTC m=+718.531261544" observedRunningTime="2026-05-25 11:34:19.263599233 +0000 UTC m=+718.912884241" watchObservedRunningTime="2026-05-25 11:34:19.265583244 +0000 UTC m=+718.914868239"
May 25 11:34:22.257174 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:22.257094    2578 generic.go:358] "Generic (PLEG): container finished" podID="241b75d1-a355-4b2b-84d2-4bf4e81ad0ac" containerID="15e09614b0ecff665d69f376a2650284b5ea403c09c4194154fa097381269730" exitCode=0
May 25 11:34:22.257549 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:22.257167    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-h56gh" event={"ID":"241b75d1-a355-4b2b-84d2-4bf4e81ad0ac","Type":"ContainerDied","Data":"15e09614b0ecff665d69f376a2650284b5ea403c09c4194154fa097381269730"}
May 25 11:34:23.393526 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:23.393503    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-h56gh"
May 25 11:34:23.495833 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:23.495792    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knsdh\" (UniqueName: \"kubernetes.io/projected/241b75d1-a355-4b2b-84d2-4bf4e81ad0ac-kube-api-access-knsdh\") pod \"241b75d1-a355-4b2b-84d2-4bf4e81ad0ac\" (UID: \"241b75d1-a355-4b2b-84d2-4bf4e81ad0ac\") "
May 25 11:34:23.498063 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:23.498036    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241b75d1-a355-4b2b-84d2-4bf4e81ad0ac-kube-api-access-knsdh" (OuterVolumeSpecName: "kube-api-access-knsdh") pod "241b75d1-a355-4b2b-84d2-4bf4e81ad0ac" (UID: "241b75d1-a355-4b2b-84d2-4bf4e81ad0ac"). InnerVolumeSpecName "kube-api-access-knsdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:34:23.596718 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:23.596626    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knsdh\" (UniqueName: \"kubernetes.io/projected/241b75d1-a355-4b2b-84d2-4bf4e81ad0ac-kube-api-access-knsdh\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:34:24.265637 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:24.265605    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-h56gh"
May 25 11:34:24.265818 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:24.265602    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-h56gh" event={"ID":"241b75d1-a355-4b2b-84d2-4bf4e81ad0ac","Type":"ContainerDied","Data":"eb5e812caca92ae0761ea5d4c4252598c62f6f1e52fed60958300447c29f69ab"}
May 25 11:34:24.265818 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:34:24.265718    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb5e812caca92ae0761ea5d4c4252598c62f6f1e52fed60958300447c29f69ab"
May 25 11:35:02.631181 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.631137    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"]
May 25 11:35:02.631625 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.631451    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241b75d1-a355-4b2b-84d2-4bf4e81ad0ac" containerName="s3-init"
May 25 11:35:02.631625 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.631484    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="241b75d1-a355-4b2b-84d2-4bf4e81ad0ac" containerName="s3-init"
May 25 11:35:02.631625 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.631571    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="241b75d1-a355-4b2b-84d2-4bf4e81ad0ac" containerName="s3-init"
May 25 11:35:02.634515 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.634498    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.639292 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.639263    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vcvlw\""
May 25 11:35:02.639428 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.639320    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:35:02.639428 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.639323    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
May 25 11:35:02.639428 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.639324    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:35:02.652139 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.652109    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"]
May 25 11:35:02.719315 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.719280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-model-cache\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.719315 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.719315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-home\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.719568 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.719335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbkx\" (UniqueName: \"kubernetes.io/projected/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kube-api-access-6nbkx\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.719568 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.719358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tls-certs\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.719568 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.719376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.719568 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.719412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-dshm\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.719568 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.719442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.820416 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.820376    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-home\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.820416 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.820420    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbkx\" (UniqueName: \"kubernetes.io/projected/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kube-api-access-6nbkx\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.820668 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.820440    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tls-certs\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.820668 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.820483    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.820668 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.820517    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-dshm\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.820668 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.820551    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.820668 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.820652    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-model-cache\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.820913 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.820837    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-home\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.820913 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.820872    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.820987 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.820966    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-model-cache\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.821031 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.821008    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.822978 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.822955    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-dshm\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.823153 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.823134    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tls-certs\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.828979 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.828956    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbkx\" (UniqueName: \"kubernetes.io/projected/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kube-api-access-6nbkx\") pod \"scheduler-inline-config-test-kserve-7574749c7-pjk7k\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:02.945036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:02.944940    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:35:03.093899 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:03.093839    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"]
May 25 11:35:03.096241 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:35:03.096213    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823ef731_7813_4ddd_9ec3_4cf50b2a224f.slice/crio-214034d41b768d48e74f09f13da8ec0a86cdfd487b9412ebdb78ffe8ab3066a8 WatchSource:0}: Error finding container 214034d41b768d48e74f09f13da8ec0a86cdfd487b9412ebdb78ffe8ab3066a8: Status 404 returned error can't find the container with id 214034d41b768d48e74f09f13da8ec0a86cdfd487b9412ebdb78ffe8ab3066a8
May 25 11:35:03.413703 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:03.413661    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k" event={"ID":"823ef731-7813-4ddd-9ec3-4cf50b2a224f","Type":"ContainerStarted","Data":"214034d41b768d48e74f09f13da8ec0a86cdfd487b9412ebdb78ffe8ab3066a8"}
May 25 11:35:07.434261 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:07.434165    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k" event={"ID":"823ef731-7813-4ddd-9ec3-4cf50b2a224f","Type":"ContainerStarted","Data":"832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b"}
May 25 11:35:53.342750 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.342670    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"]
May 25 11:35:53.345798 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.345781    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.348883 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.348862    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\""
May 25 11:35:53.364239 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.364214    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"]
May 25 11:35:53.415255 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.415219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.415255 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.415260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.415448 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.415278    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.415448 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.415362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.415448 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.415400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.415448 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.415420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6425c\" (UniqueName: \"kubernetes.io/projected/03e7a84a-9068-4184-8faf-86ab22c86112-kube-api-access-6425c\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.415606 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.415514    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e7a84a-9068-4184-8faf-86ab22c86112-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.516385 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.516342    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.516385 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.516391    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.516642 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.516509    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6425c\" (UniqueName: \"kubernetes.io/projected/03e7a84a-9068-4184-8faf-86ab22c86112-kube-api-access-6425c\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.516642 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.516584    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e7a84a-9068-4184-8faf-86ab22c86112-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.516721 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.516652    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.516721 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.516688    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.516823 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.516719    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.516823 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.516766    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.516957 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.516820    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.517064 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.517043    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.517119 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.517062    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.519164 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.519145    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.519247 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.519219    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e7a84a-9068-4184-8faf-86ab22c86112-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.525751 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.525727    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6425c\" (UniqueName: \"kubernetes.io/projected/03e7a84a-9068-4184-8faf-86ab22c86112-kube-api-access-6425c\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.656504 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.656398    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:35:53.811408 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:53.810427    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"]
May 25 11:35:53.813736 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:35:53.813705    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e7a84a_9068_4184_8faf_86ab22c86112.slice/crio-33457be34c7fc3544c4eda9a1222d10c7a76875cd3db1224e9eb9480cd0a78da WatchSource:0}: Error finding container 33457be34c7fc3544c4eda9a1222d10c7a76875cd3db1224e9eb9480cd0a78da: Status 404 returned error can't find the container with id 33457be34c7fc3544c4eda9a1222d10c7a76875cd3db1224e9eb9480cd0a78da
May 25 11:35:54.607074 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:54.607036    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz" event={"ID":"03e7a84a-9068-4184-8faf-86ab22c86112","Type":"ContainerStarted","Data":"cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc"}
May 25 11:35:54.607074 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:35:54.607075    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz" event={"ID":"03e7a84a-9068-4184-8faf-86ab22c86112","Type":"ContainerStarted","Data":"33457be34c7fc3544c4eda9a1222d10c7a76875cd3db1224e9eb9480cd0a78da"}
May 25 11:36:22.712271 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:22.712234    2578 generic.go:358] "Generic (PLEG): container finished" podID="823ef731-7813-4ddd-9ec3-4cf50b2a224f" containerID="832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b" exitCode=0
May 25 11:36:22.712717 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:22.712295    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k" event={"ID":"823ef731-7813-4ddd-9ec3-4cf50b2a224f","Type":"ContainerDied","Data":"832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b"}
May 25 11:36:24.722575 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:24.722530    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k" event={"ID":"823ef731-7813-4ddd-9ec3-4cf50b2a224f","Type":"ContainerStarted","Data":"6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d"}
May 25 11:36:24.767945 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:24.767888    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k" podStartSLOduration=2.116720922 podStartE2EDuration="1m22.767872592s" podCreationTimestamp="2026-05-25 11:35:02 +0000 UTC" firstStartedPulling="2026-05-25 11:35:03.098234495 +0000 UTC m=+762.747519488" lastFinishedPulling="2026-05-25 11:36:23.749386186 +0000 UTC m=+843.398671158" observedRunningTime="2026-05-25 11:36:24.766110463 +0000 UTC m=+844.415395449" watchObservedRunningTime="2026-05-25 11:36:24.767872592 +0000 UTC m=+844.417157586"
May 25 11:36:32.945792 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:32.945755    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:36:32.946294 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:32.945836    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:36:32.958943 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:32.958911    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:36:33.771002 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:33.770971    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:36:35.737603 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:35.737566    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"]
May 25 11:36:35.776092 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:36:35.776053    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs: secret "scheduler-inline-config-test-kserve-self-signed-certs" not found
May 25 11:36:35.776290 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:36:35.776133    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tls-certs podName:823ef731-7813-4ddd-9ec3-4cf50b2a224f nodeName:}" failed. No retries permitted until 2026-05-25 11:36:36.276116898 +0000 UTC m=+855.925401871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tls-certs") pod "scheduler-inline-config-test-kserve-7574749c7-pjk7k" (UID: "823ef731-7813-4ddd-9ec3-4cf50b2a224f") : secret "scheduler-inline-config-test-kserve-self-signed-certs" not found
May 25 11:36:36.281171 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:36:36.281131    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs: secret "scheduler-inline-config-test-kserve-self-signed-certs" not found
May 25 11:36:36.281358 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:36:36.281203    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tls-certs podName:823ef731-7813-4ddd-9ec3-4cf50b2a224f nodeName:}" failed. No retries permitted until 2026-05-25 11:36:37.2811873 +0000 UTC m=+856.930472273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tls-certs") pod "scheduler-inline-config-test-kserve-7574749c7-pjk7k" (UID: "823ef731-7813-4ddd-9ec3-4cf50b2a224f") : secret "scheduler-inline-config-test-kserve-self-signed-certs" not found
May 25 11:36:36.769216 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:36.769173    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k" podUID="823ef731-7813-4ddd-9ec3-4cf50b2a224f" containerName="main" containerID="cri-o://6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d" gracePeriod=30
May 25 11:36:37.024396 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.024334    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:36:37.087269 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.087236    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tmp-dir\") pod \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") "
May 25 11:36:37.087422 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.087285    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-model-cache\") pod \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") "
May 25 11:36:37.087422 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.087315    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nbkx\" (UniqueName: \"kubernetes.io/projected/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kube-api-access-6nbkx\") pod \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") "
May 25 11:36:37.087422 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.087350    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-dshm\") pod \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") "
May 25 11:36:37.087422 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.087396    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kserve-provision-location\") pod \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") "
May 25 11:36:37.087631 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.087429    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tls-certs\") pod \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") "
May 25 11:36:37.087631 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.087491    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-home\") pod \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\" (UID: \"823ef731-7813-4ddd-9ec3-4cf50b2a224f\") "
May 25 11:36:37.087631 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.087527    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "823ef731-7813-4ddd-9ec3-4cf50b2a224f" (UID: "823ef731-7813-4ddd-9ec3-4cf50b2a224f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:36:37.087768 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.087645    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-model-cache" (OuterVolumeSpecName: "model-cache") pod "823ef731-7813-4ddd-9ec3-4cf50b2a224f" (UID: "823ef731-7813-4ddd-9ec3-4cf50b2a224f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:36:37.088156 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.087907    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:36:37.088156 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.087933    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:36:37.088156 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.088010    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-home" (OuterVolumeSpecName: "home") pod "823ef731-7813-4ddd-9ec3-4cf50b2a224f" (UID: "823ef731-7813-4ddd-9ec3-4cf50b2a224f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:36:37.090010 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.089983    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "823ef731-7813-4ddd-9ec3-4cf50b2a224f" (UID: "823ef731-7813-4ddd-9ec3-4cf50b2a224f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:36:37.090134 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.090117    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-dshm" (OuterVolumeSpecName: "dshm") pod "823ef731-7813-4ddd-9ec3-4cf50b2a224f" (UID: "823ef731-7813-4ddd-9ec3-4cf50b2a224f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:36:37.090134 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.090122    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kube-api-access-6nbkx" (OuterVolumeSpecName: "kube-api-access-6nbkx") pod "823ef731-7813-4ddd-9ec3-4cf50b2a224f" (UID: "823ef731-7813-4ddd-9ec3-4cf50b2a224f"). InnerVolumeSpecName "kube-api-access-6nbkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:36:37.141354 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.141304    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "823ef731-7813-4ddd-9ec3-4cf50b2a224f" (UID: "823ef731-7813-4ddd-9ec3-4cf50b2a224f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:36:37.188292 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.188261    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbkx\" (UniqueName: \"kubernetes.io/projected/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kube-api-access-6nbkx\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:36:37.188292 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.188288    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:36:37.188493 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.188300    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:36:37.188493 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.188309    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/823ef731-7813-4ddd-9ec3-4cf50b2a224f-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:36:37.188493 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.188318    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/823ef731-7813-4ddd-9ec3-4cf50b2a224f-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:36:37.774531 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.774497    2578 generic.go:358] "Generic (PLEG): container finished" podID="823ef731-7813-4ddd-9ec3-4cf50b2a224f" containerID="6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d" exitCode=0
May 25 11:36:37.775017 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.774592    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k" event={"ID":"823ef731-7813-4ddd-9ec3-4cf50b2a224f","Type":"ContainerDied","Data":"6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d"}
May 25 11:36:37.775017 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.774618    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k" event={"ID":"823ef731-7813-4ddd-9ec3-4cf50b2a224f","Type":"ContainerDied","Data":"214034d41b768d48e74f09f13da8ec0a86cdfd487b9412ebdb78ffe8ab3066a8"}
May 25 11:36:37.775017 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.774633    2578 scope.go:117] "RemoveContainer" containerID="6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d"
May 25 11:36:37.775017 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.774654    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"
May 25 11:36:37.783878 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.783860    2578 scope.go:117] "RemoveContainer" containerID="832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b"
May 25 11:36:37.804550 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.804503    2578 scope.go:117] "RemoveContainer" containerID="6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d"
May 25 11:36:37.804854 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:36:37.804833    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d\": container with ID starting with 6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d not found: ID does not exist" containerID="6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d"
May 25 11:36:37.804917 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.804861    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d"} err="failed to get container status \"6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d\": rpc error: code = NotFound desc = could not find container \"6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d\": container with ID starting with 6fcba3d0475dfc3d65227c79c50df2173318b499924566f938f7556233a7017d not found: ID does not exist"
May 25 11:36:37.804917 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.804881    2578 scope.go:117] "RemoveContainer" containerID="832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b"
May 25 11:36:37.805070 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.805048    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"]
May 25 11:36:37.805116 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:36:37.805088    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b\": container with ID starting with 832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b not found: ID does not exist" containerID="832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b"
May 25 11:36:37.805152 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.805121    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b"} err="failed to get container status \"832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b\": rpc error: code = NotFound desc = could not find container \"832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b\": container with ID starting with 832efc224821457950839ff0966cd6e212d0a413b0b0cd28c7e8fecaa5b01d5b not found: ID does not exist"
May 25 11:36:37.810280 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:37.810257    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7574749c7-pjk7k"]
May 25 11:36:38.935332 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:36:38.935299    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823ef731-7813-4ddd-9ec3-4cf50b2a224f" path="/var/lib/kubelet/pods/823ef731-7813-4ddd-9ec3-4cf50b2a224f/volumes"
May 25 11:37:20.921841 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:20.921758    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:37:20.924344 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:20.924316    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:37:32.072597 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.072563    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"]
May 25 11:37:32.073063 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.072864    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823ef731-7813-4ddd-9ec3-4cf50b2a224f" containerName="main"
May 25 11:37:32.073063 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.072874    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="823ef731-7813-4ddd-9ec3-4cf50b2a224f" containerName="main"
May 25 11:37:32.073063 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.072890    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823ef731-7813-4ddd-9ec3-4cf50b2a224f" containerName="storage-initializer"
May 25 11:37:32.073063 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.072896    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="823ef731-7813-4ddd-9ec3-4cf50b2a224f" containerName="storage-initializer"
May 25 11:37:32.073063 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.072953    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="823ef731-7813-4ddd-9ec3-4cf50b2a224f" containerName="main"
May 25 11:37:32.077204 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.077181    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.080852 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.080829    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
May 25 11:37:32.092139 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.092116    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"]
May 25 11:37:32.140726 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.140687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.140726 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.140727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.140980 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.140755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.140980 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.140840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvp2x\" (UniqueName: \"kubernetes.io/projected/f4ff8b98-f459-454f-8975-8523b1607fe6-kube-api-access-vvp2x\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.140980 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.140917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.140980 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.140963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-dshm\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.141136 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.140990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-home\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.241564 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.241524    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.241758 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.241578    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-dshm\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.241758 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.241596    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-home\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.241758 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.241613    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.241758 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.241725    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.241951 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.241788    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.241951 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.241845    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vvp2x\" (UniqueName: \"kubernetes.io/projected/f4ff8b98-f459-454f-8975-8523b1607fe6-kube-api-access-vvp2x\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.242055 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.241966    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.242055 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.242005    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-model-cache\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.242137 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.242050    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-home\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.242171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.242153    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.244087 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.244065    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-dshm\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.244432 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.244413    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.265007 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.264977    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvp2x\" (UniqueName: \"kubernetes.io/projected/f4ff8b98-f459-454f-8975-8523b1607fe6-kube-api-access-vvp2x\") pod \"scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.387724 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.387625    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:37:32.545974 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.545942    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"]
May 25 11:37:32.547599 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:37:32.547569    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ff8b98_f459_454f_8975_8523b1607fe6.slice/crio-7aafd640fbe94794e2c21272b8ac44839a2b26a2b74ef377df18a4d8fa94faef WatchSource:0}: Error finding container 7aafd640fbe94794e2c21272b8ac44839a2b26a2b74ef377df18a4d8fa94faef: Status 404 returned error can't find the container with id 7aafd640fbe94794e2c21272b8ac44839a2b26a2b74ef377df18a4d8fa94faef
May 25 11:37:32.969268 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.969231    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv" event={"ID":"f4ff8b98-f459-454f-8975-8523b1607fe6","Type":"ContainerStarted","Data":"c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf"}
May 25 11:37:32.969268 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:32.969273    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv" event={"ID":"f4ff8b98-f459-454f-8975-8523b1607fe6","Type":"ContainerStarted","Data":"7aafd640fbe94794e2c21272b8ac44839a2b26a2b74ef377df18a4d8fa94faef"}
May 25 11:37:42.003226 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:42.003189    2578 generic.go:358] "Generic (PLEG): container finished" podID="03e7a84a-9068-4184-8faf-86ab22c86112" containerID="cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc" exitCode=0
May 25 11:37:42.003645 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:42.003265    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz" event={"ID":"03e7a84a-9068-4184-8faf-86ab22c86112","Type":"ContainerDied","Data":"cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc"}
May 25 11:37:43.008446 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:43.008410    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz" event={"ID":"03e7a84a-9068-4184-8faf-86ab22c86112","Type":"ContainerStarted","Data":"4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7"}
May 25 11:37:43.062686 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:43.062632    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz" podStartSLOduration=110.062618376 podStartE2EDuration="1m50.062618376s" podCreationTimestamp="2026-05-25 11:35:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:37:43.061893886 +0000 UTC m=+922.711178881" watchObservedRunningTime="2026-05-25 11:37:43.062618376 +0000 UTC m=+922.711903389"
May 25 11:37:43.656883 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:43.656841    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:37:43.656883 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:43.656889    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:37:43.669476 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:43.669424    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:37:44.022917 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:44.022889    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:37:50.545595 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.545555    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"]
May 25 11:37:50.546986 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.545821    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz" podUID="03e7a84a-9068-4184-8faf-86ab22c86112" containerName="main" containerID="cri-o://4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7" gracePeriod=30
May 25 11:37:50.797028 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.796967    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:37:50.897882 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.897852    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-tmp-dir\") pod \"03e7a84a-9068-4184-8faf-86ab22c86112\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") "
May 25 11:37:50.898075 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.897904    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-model-cache\") pod \"03e7a84a-9068-4184-8faf-86ab22c86112\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") "
May 25 11:37:50.898075 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.897928    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e7a84a-9068-4184-8faf-86ab22c86112-tls-certs\") pod \"03e7a84a-9068-4184-8faf-86ab22c86112\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") "
May 25 11:37:50.898075 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.897972    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6425c\" (UniqueName: \"kubernetes.io/projected/03e7a84a-9068-4184-8faf-86ab22c86112-kube-api-access-6425c\") pod \"03e7a84a-9068-4184-8faf-86ab22c86112\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") "
May 25 11:37:50.898075 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.897994    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-dshm\") pod \"03e7a84a-9068-4184-8faf-86ab22c86112\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") "
May 25 11:37:50.898075 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.898036    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-home\") pod \"03e7a84a-9068-4184-8faf-86ab22c86112\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") "
May 25 11:37:50.898075 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.898071    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-kserve-provision-location\") pod \"03e7a84a-9068-4184-8faf-86ab22c86112\" (UID: \"03e7a84a-9068-4184-8faf-86ab22c86112\") "
May 25 11:37:50.898381 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.898205    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03e7a84a-9068-4184-8faf-86ab22c86112" (UID: "03e7a84a-9068-4184-8faf-86ab22c86112"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:37:50.898381 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.898231    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-model-cache" (OuterVolumeSpecName: "model-cache") pod "03e7a84a-9068-4184-8faf-86ab22c86112" (UID: "03e7a84a-9068-4184-8faf-86ab22c86112"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:37:50.898381 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.898355    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-home" (OuterVolumeSpecName: "home") pod "03e7a84a-9068-4184-8faf-86ab22c86112" (UID: "03e7a84a-9068-4184-8faf-86ab22c86112"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:37:50.898567 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.898411    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:37:50.898567 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.898431    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:37:50.900386 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.900354    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e7a84a-9068-4184-8faf-86ab22c86112-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03e7a84a-9068-4184-8faf-86ab22c86112" (UID: "03e7a84a-9068-4184-8faf-86ab22c86112"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:37:50.900533 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.900382    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-dshm" (OuterVolumeSpecName: "dshm") pod "03e7a84a-9068-4184-8faf-86ab22c86112" (UID: "03e7a84a-9068-4184-8faf-86ab22c86112"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:37:50.900533 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.900486    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e7a84a-9068-4184-8faf-86ab22c86112-kube-api-access-6425c" (OuterVolumeSpecName: "kube-api-access-6425c") pod "03e7a84a-9068-4184-8faf-86ab22c86112" (UID: "03e7a84a-9068-4184-8faf-86ab22c86112"). InnerVolumeSpecName "kube-api-access-6425c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:37:50.953070 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.953030    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03e7a84a-9068-4184-8faf-86ab22c86112" (UID: "03e7a84a-9068-4184-8faf-86ab22c86112"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:37:50.999768 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.999732    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e7a84a-9068-4184-8faf-86ab22c86112-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:37:50.999768 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.999761    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6425c\" (UniqueName: \"kubernetes.io/projected/03e7a84a-9068-4184-8faf-86ab22c86112-kube-api-access-6425c\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:37:50.999768 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.999772    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:37:51.000069 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.999781    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:37:51.000069 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:50.999789    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03e7a84a-9068-4184-8faf-86ab22c86112-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:37:51.037032 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.037001    2578 generic.go:358] "Generic (PLEG): container finished" podID="03e7a84a-9068-4184-8faf-86ab22c86112" containerID="4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7" exitCode=0
May 25 11:37:51.037190 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.037080    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"
May 25 11:37:51.037190 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.037079    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz" event={"ID":"03e7a84a-9068-4184-8faf-86ab22c86112","Type":"ContainerDied","Data":"4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7"}
May 25 11:37:51.037273 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.037192    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz" event={"ID":"03e7a84a-9068-4184-8faf-86ab22c86112","Type":"ContainerDied","Data":"33457be34c7fc3544c4eda9a1222d10c7a76875cd3db1224e9eb9480cd0a78da"}
May 25 11:37:51.037273 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.037209    2578 scope.go:117] "RemoveContainer" containerID="4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7"
May 25 11:37:51.047025 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.047006    2578 scope.go:117] "RemoveContainer" containerID="cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc"
May 25 11:37:51.057392 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.057368    2578 scope.go:117] "RemoveContainer" containerID="4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7"
May 25 11:37:51.057787 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:37:51.057762    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7\": container with ID starting with 4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7 not found: ID does not exist" containerID="4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7"
May 25 11:37:51.057891 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.057797    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7"} err="failed to get container status \"4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7\": rpc error: code = NotFound desc = could not find container \"4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7\": container with ID starting with 4d38212bba78512d22cd6ce279e0295565d3c4f8561663e4b16ef3bbeedbb8c7 not found: ID does not exist"
May 25 11:37:51.057891 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.057821    2578 scope.go:117] "RemoveContainer" containerID="cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc"
May 25 11:37:51.058350 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:37:51.058097    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc\": container with ID starting with cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc not found: ID does not exist" containerID="cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc"
May 25 11:37:51.058350 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.058153    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc"} err="failed to get container status \"cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc\": rpc error: code = NotFound desc = could not find container \"cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc\": container with ID starting with cba4e9f8b059a9d24493f995d1ec08c69bfd5360987412196b353b0e6c8f07dc not found: ID does not exist"
May 25 11:37:51.077628 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.077599    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"]
May 25 11:37:51.093237 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:51.093207    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-79d9b6d78f6fjfz"]
May 25 11:37:52.935740 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:37:52.935706    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03e7a84a-9068-4184-8faf-86ab22c86112" path="/var/lib/kubelet/pods/03e7a84a-9068-4184-8faf-86ab22c86112/volumes"
May 25 11:38:04.138936 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.138886    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"]
May 25 11:38:04.139394 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.139198    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e7a84a-9068-4184-8faf-86ab22c86112" containerName="main"
May 25 11:38:04.139394 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.139209    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e7a84a-9068-4184-8faf-86ab22c86112" containerName="main"
May 25 11:38:04.139394 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.139218    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e7a84a-9068-4184-8faf-86ab22c86112" containerName="storage-initializer"
May 25 11:38:04.139394 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.139224    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e7a84a-9068-4184-8faf-86ab22c86112" containerName="storage-initializer"
May 25 11:38:04.139394 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.139277    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="03e7a84a-9068-4184-8faf-86ab22c86112" containerName="main"
May 25 11:38:04.142808 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.142786    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.145758 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.145741    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 25 11:38:04.168549 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.168517    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"]
May 25 11:38:04.209805 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.209760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.209805 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.209808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.210052 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.209855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.210052 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.209889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.210052 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.209953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.210052 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.210012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcwks\" (UniqueName: \"kubernetes.io/projected/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kube-api-access-vcwks\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.210199 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.210079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.311322 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.311287    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.311542 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.311350    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.311542 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.311379    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.311542 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.311417    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.311542 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.311440    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.311542 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.311505    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.311805 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.311554    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwks\" (UniqueName: \"kubernetes.io/projected/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kube-api-access-vcwks\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.311861 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.311836    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.311915 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.311861    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.311956 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.311913    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.311956 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.311935    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.313879 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.313858    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.314123 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.314105    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.328125 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.328095    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcwks\" (UniqueName: \"kubernetes.io/projected/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kube-api-access-vcwks\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.452575 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.452452    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:04.598916 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.598886    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"]
May 25 11:38:04.600853 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:38:04.600813    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4ab078_14d3_4c70_ab2a_eaf6bb4439c6.slice/crio-8b2072f5f1c9f7e8c49b3f3ab0e0ae65f7ac05aa0d6a065438d4494953cd6131 WatchSource:0}: Error finding container 8b2072f5f1c9f7e8c49b3f3ab0e0ae65f7ac05aa0d6a065438d4494953cd6131: Status 404 returned error can't find the container with id 8b2072f5f1c9f7e8c49b3f3ab0e0ae65f7ac05aa0d6a065438d4494953cd6131
May 25 11:38:04.602624 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:04.602603    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:38:05.096060 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:05.096021    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" event={"ID":"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6","Type":"ContainerStarted","Data":"1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1"}
May 25 11:38:05.096060 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:05.096058    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" event={"ID":"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6","Type":"ContainerStarted","Data":"8b2072f5f1c9f7e8c49b3f3ab0e0ae65f7ac05aa0d6a065438d4494953cd6131"}
May 25 11:38:12.124089 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:12.124000    2578 generic.go:358] "Generic (PLEG): container finished" podID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerID="1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1" exitCode=0
May 25 11:38:12.124089 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:12.124042    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" event={"ID":"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6","Type":"ContainerDied","Data":"1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1"}
May 25 11:38:41.253438 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:41.253400    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" event={"ID":"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6","Type":"ContainerStarted","Data":"bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b"}
May 25 11:38:44.453167 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:44.453123    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:44.453697 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:44.453233    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:38:44.454865 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:44.454838    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 11:38:54.453175 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:38:54.453121    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 11:39:04.453914 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:04.453863    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 11:39:14.453419 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:14.453369    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 11:39:24.453353 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:24.453297    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 11:39:34.453108 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:34.453064    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 11:39:39.473617 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:39.473586    2578 generic.go:358] "Generic (PLEG): container finished" podID="f4ff8b98-f459-454f-8975-8523b1607fe6" containerID="c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf" exitCode=0
May 25 11:39:39.474005 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:39.473666    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv" event={"ID":"f4ff8b98-f459-454f-8975-8523b1607fe6","Type":"ContainerDied","Data":"c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf"}
May 25 11:39:39.499731 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:39.499668    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podStartSLOduration=66.734630782 podStartE2EDuration="1m35.499648734s" podCreationTimestamp="2026-05-25 11:38:04 +0000 UTC" firstStartedPulling="2026-05-25 11:38:12.125159667 +0000 UTC m=+951.774444641" lastFinishedPulling="2026-05-25 11:38:40.89017762 +0000 UTC m=+980.539462593" observedRunningTime="2026-05-25 11:38:41.305830659 +0000 UTC m=+980.955115653" watchObservedRunningTime="2026-05-25 11:39:39.499648734 +0000 UTC m=+1039.148933730"
May 25 11:39:40.479568 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:40.479531    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv" event={"ID":"f4ff8b98-f459-454f-8975-8523b1607fe6","Type":"ContainerStarted","Data":"56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433"}
May 25 11:39:40.510480 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:40.510406    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv" podStartSLOduration=128.510388728 podStartE2EDuration="2m8.510388728s" podCreationTimestamp="2026-05-25 11:37:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:39:40.508391677 +0000 UTC m=+1040.157676672" watchObservedRunningTime="2026-05-25 11:39:40.510388728 +0000 UTC m=+1040.159673723"
May 25 11:39:42.388279 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:42.388234    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:39:42.388738 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:42.388297    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:39:42.401125 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:42.401100    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:39:42.498704 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:42.498668    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:39:43.581223 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:43.581191    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"]
May 25 11:39:43.595320 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:39:43.595284    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs: secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
May 25 11:39:43.595513 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:39:43.595367    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs podName:f4ff8b98-f459-454f-8975-8523b1607fe6 nodeName:}" failed. No retries permitted until 2026-05-25 11:39:44.095347456 +0000 UTC m=+1043.744632428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs") pod "scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv" (UID: "f4ff8b98-f459-454f-8975-8523b1607fe6") : secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
May 25 11:39:44.101029 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:39:44.100988    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs: secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
May 25 11:39:44.101221 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:39:44.101073    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs podName:f4ff8b98-f459-454f-8975-8523b1607fe6 nodeName:}" failed. No retries permitted until 2026-05-25 11:39:45.101054278 +0000 UTC m=+1044.750339251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs") pod "scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv" (UID: "f4ff8b98-f459-454f-8975-8523b1607fe6") : secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
May 25 11:39:44.453787 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:44.453681    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 11:39:45.110677 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:39:45.110642    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-self-signed-certs: secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
May 25 11:39:45.111135 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:39:45.110718    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs podName:f4ff8b98-f459-454f-8975-8523b1607fe6 nodeName:}" failed. No retries permitted until 2026-05-25 11:39:47.110702674 +0000 UTC m=+1046.759987647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs") pod "scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv" (UID: "f4ff8b98-f459-454f-8975-8523b1607fe6") : secret "scheduler-ha-replicas-test-kserve-self-signed-certs" not found
May 25 11:39:45.498313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.498255    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv" podUID="f4ff8b98-f459-454f-8975-8523b1607fe6" containerName="main" containerID="cri-o://56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433" gracePeriod=30
May 25 11:39:45.771265 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.771240    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:39:45.917858 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.917815    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-model-cache\") pod \"f4ff8b98-f459-454f-8975-8523b1607fe6\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") "
May 25 11:39:45.917858 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.917862    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-kserve-provision-location\") pod \"f4ff8b98-f459-454f-8975-8523b1607fe6\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") "
May 25 11:39:45.918091 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.917887    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs\") pod \"f4ff8b98-f459-454f-8975-8523b1607fe6\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") "
May 25 11:39:45.918091 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.917906    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvp2x\" (UniqueName: \"kubernetes.io/projected/f4ff8b98-f459-454f-8975-8523b1607fe6-kube-api-access-vvp2x\") pod \"f4ff8b98-f459-454f-8975-8523b1607fe6\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") "
May 25 11:39:45.918091 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.917927    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-tmp-dir\") pod \"f4ff8b98-f459-454f-8975-8523b1607fe6\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") "
May 25 11:39:45.918252 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.918085    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-home\") pod \"f4ff8b98-f459-454f-8975-8523b1607fe6\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") "
May 25 11:39:45.918252 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.918162    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-dshm\") pod \"f4ff8b98-f459-454f-8975-8523b1607fe6\" (UID: \"f4ff8b98-f459-454f-8975-8523b1607fe6\") "
May 25 11:39:45.918252 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.918196    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4ff8b98-f459-454f-8975-8523b1607fe6" (UID: "f4ff8b98-f459-454f-8975-8523b1607fe6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:39:45.918252 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.918207    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4ff8b98-f459-454f-8975-8523b1607fe6" (UID: "f4ff8b98-f459-454f-8975-8523b1607fe6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:39:45.918440 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.918365    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-home" (OuterVolumeSpecName: "home") pod "f4ff8b98-f459-454f-8975-8523b1607fe6" (UID: "f4ff8b98-f459-454f-8975-8523b1607fe6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:39:45.918544 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.918480    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:39:45.918544 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.918498    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:39:45.918544 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.918507    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:39:45.920379 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.920345    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ff8b98-f459-454f-8975-8523b1607fe6-kube-api-access-vvp2x" (OuterVolumeSpecName: "kube-api-access-vvp2x") pod "f4ff8b98-f459-454f-8975-8523b1607fe6" (UID: "f4ff8b98-f459-454f-8975-8523b1607fe6"). InnerVolumeSpecName "kube-api-access-vvp2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:39:45.920547 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.920401    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4ff8b98-f459-454f-8975-8523b1607fe6" (UID: "f4ff8b98-f459-454f-8975-8523b1607fe6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:39:45.920547 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.920428    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-dshm" (OuterVolumeSpecName: "dshm") pod "f4ff8b98-f459-454f-8975-8523b1607fe6" (UID: "f4ff8b98-f459-454f-8975-8523b1607fe6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:39:45.987400 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:45.987357    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4ff8b98-f459-454f-8975-8523b1607fe6" (UID: "f4ff8b98-f459-454f-8975-8523b1607fe6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:39:46.019244 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.019164    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:39:46.019244 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.019198    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ff8b98-f459-454f-8975-8523b1607fe6-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:39:46.019244 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.019210    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvp2x\" (UniqueName: \"kubernetes.io/projected/f4ff8b98-f459-454f-8975-8523b1607fe6-kube-api-access-vvp2x\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:39:46.019244 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.019221    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ff8b98-f459-454f-8975-8523b1607fe6-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:39:46.503754 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.503724    2578 generic.go:358] "Generic (PLEG): container finished" podID="f4ff8b98-f459-454f-8975-8523b1607fe6" containerID="56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433" exitCode=0
May 25 11:39:46.504166 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.503791    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"
May 25 11:39:46.504166 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.503811    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv" event={"ID":"f4ff8b98-f459-454f-8975-8523b1607fe6","Type":"ContainerDied","Data":"56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433"}
May 25 11:39:46.504166 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.503861    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv" event={"ID":"f4ff8b98-f459-454f-8975-8523b1607fe6","Type":"ContainerDied","Data":"7aafd640fbe94794e2c21272b8ac44839a2b26a2b74ef377df18a4d8fa94faef"}
May 25 11:39:46.504166 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.503880    2578 scope.go:117] "RemoveContainer" containerID="56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433"
May 25 11:39:46.513581 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.513558    2578 scope.go:117] "RemoveContainer" containerID="c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf"
May 25 11:39:46.529526 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.529497    2578 scope.go:117] "RemoveContainer" containerID="56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433"
May 25 11:39:46.529904 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:39:46.529881    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433\": container with ID starting with 56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433 not found: ID does not exist" containerID="56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433"
May 25 11:39:46.530025 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.529919    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433"} err="failed to get container status \"56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433\": rpc error: code = NotFound desc = could not find container \"56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433\": container with ID starting with 56575409092cae33c432fc9ef14ac5b38a0b0819a88043ac5339c49324d06433 not found: ID does not exist"
May 25 11:39:46.530025 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.529944    2578 scope.go:117] "RemoveContainer" containerID="c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf"
May 25 11:39:46.530407 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:39:46.530382    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf\": container with ID starting with c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf not found: ID does not exist" containerID="c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf"
May 25 11:39:46.530535 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.530414    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf"} err="failed to get container status \"c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf\": rpc error: code = NotFound desc = could not find container \"c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf\": container with ID starting with c85d1425ba3fe5e5ab968c57b697f2595e82a20e606a52933816e16c6f7acecf not found: ID does not exist"
May 25 11:39:46.533020 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.532987    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"]
May 25 11:39:46.542563 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.542534    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-9c7865cd9-8s5dv"]
May 25 11:39:46.934981 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:46.934898    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4ff8b98-f459-454f-8975-8523b1607fe6" path="/var/lib/kubelet/pods/f4ff8b98-f459-454f-8975-8523b1607fe6/volumes"
May 25 11:39:49.264360 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.264324    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"]
May 25 11:39:49.264849 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.264638    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ff8b98-f459-454f-8975-8523b1607fe6" containerName="storage-initializer"
May 25 11:39:49.264849 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.264650    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ff8b98-f459-454f-8975-8523b1607fe6" containerName="storage-initializer"
May 25 11:39:49.264849 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.264658    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ff8b98-f459-454f-8975-8523b1607fe6" containerName="main"
May 25 11:39:49.264849 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.264664    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ff8b98-f459-454f-8975-8523b1607fe6" containerName="main"
May 25 11:39:49.264849 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.264714    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="f4ff8b98-f459-454f-8975-8523b1607fe6" containerName="main"
May 25 11:39:49.267729 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.267705    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.271436 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.271407    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
May 25 11:39:49.288353 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.288314    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"]
May 25 11:39:49.346407 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.346372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-home\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.346630 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.346420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-dshm\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.346630 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.346480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.346630 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.346560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tls-certs\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.346630 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.346592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-model-cache\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.346872 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.346660    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.346872 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.346700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47frw\" (UniqueName: \"kubernetes.io/projected/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kube-api-access-47frw\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.447562 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.447516    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-home\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.447562 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.447563    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-dshm\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.447836 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.447600    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.447836 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.447627    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tls-certs\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.447836 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.447648    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-model-cache\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.447836 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.447811    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.448400 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.447871    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-47frw\" (UniqueName: \"kubernetes.io/projected/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kube-api-access-47frw\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.448400 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.447953    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-home\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.448400 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.447987    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tmp-dir\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.448400 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.448128    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.448400 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.448198    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-model-cache\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.450035 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.450011    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-dshm\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.450295 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.450274    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tls-certs\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.459808 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.459776    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47frw\" (UniqueName: \"kubernetes.io/projected/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kube-api-access-47frw\") pod \"precise-prefix-cache-test-kserve-55c6c598f5-44n6w\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.579302 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.579220    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:49.729185 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:49.729151    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"]
May 25 11:39:49.730770 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:39:49.730746    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60befab0_c2f8_4c23_8e2c_9aced50fdc3d.slice/crio-2f15bb49e50907452dfd140004c3fe35a55edfbb268ab67d9559ec322dcb9607 WatchSource:0}: Error finding container 2f15bb49e50907452dfd140004c3fe35a55edfbb268ab67d9559ec322dcb9607: Status 404 returned error can't find the container with id 2f15bb49e50907452dfd140004c3fe35a55edfbb268ab67d9559ec322dcb9607
May 25 11:39:50.521594 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:50.521556    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w" event={"ID":"60befab0-c2f8-4c23-8e2c-9aced50fdc3d","Type":"ContainerStarted","Data":"83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0"}
May 25 11:39:50.521594 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:50.521597    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w" event={"ID":"60befab0-c2f8-4c23-8e2c-9aced50fdc3d","Type":"ContainerStarted","Data":"2f15bb49e50907452dfd140004c3fe35a55edfbb268ab67d9559ec322dcb9607"}
May 25 11:39:54.453916 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:54.453865    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 11:39:54.541169 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:54.541131    2578 generic.go:358] "Generic (PLEG): container finished" podID="60befab0-c2f8-4c23-8e2c-9aced50fdc3d" containerID="83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0" exitCode=0
May 25 11:39:54.541331 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:54.541203    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w" event={"ID":"60befab0-c2f8-4c23-8e2c-9aced50fdc3d","Type":"ContainerDied","Data":"83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0"}
May 25 11:39:55.547986 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:55.547944    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w" event={"ID":"60befab0-c2f8-4c23-8e2c-9aced50fdc3d","Type":"ContainerStarted","Data":"4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197"}
May 25 11:39:55.576132 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:55.576083    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w" podStartSLOduration=6.576066987 podStartE2EDuration="6.576066987s" podCreationTimestamp="2026-05-25 11:39:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:39:55.572768568 +0000 UTC m=+1055.222053564" watchObservedRunningTime="2026-05-25 11:39:55.576066987 +0000 UTC m=+1055.225351982"
May 25 11:39:59.579768 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:59.579715    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:59.579768 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:59.579759    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:39:59.592649 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:39:59.592616    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:40:00.577717 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:00.577682    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:40:04.453374 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:04.453330    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 11:40:14.453392 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:14.453277    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
May 25 11:40:23.677376 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:23.677342    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"]
May 25 11:40:23.677810 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:23.677697    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w" podUID="60befab0-c2f8-4c23-8e2c-9aced50fdc3d" containerName="main" containerID="cri-o://4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197" gracePeriod=30
May 25 11:40:23.924725 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:23.924689    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:40:24.049479 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.049423    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-home\") pod \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") "
May 25 11:40:24.049672 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.049503    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tls-certs\") pod \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") "
May 25 11:40:24.049672 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.049558    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47frw\" (UniqueName: \"kubernetes.io/projected/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kube-api-access-47frw\") pod \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") "
May 25 11:40:24.049672 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.049579    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-dshm\") pod \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") "
May 25 11:40:24.049672 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.049597    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-model-cache\") pod \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") "
May 25 11:40:24.049672 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.049660    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kserve-provision-location\") pod \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") "
May 25 11:40:24.049924 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.049702    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tmp-dir\") pod \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\" (UID: \"60befab0-c2f8-4c23-8e2c-9aced50fdc3d\") "
May 25 11:40:24.049924 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.049761    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-home" (OuterVolumeSpecName: "home") pod "60befab0-c2f8-4c23-8e2c-9aced50fdc3d" (UID: "60befab0-c2f8-4c23-8e2c-9aced50fdc3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:24.050025 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.049934    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "60befab0-c2f8-4c23-8e2c-9aced50fdc3d" (UID: "60befab0-c2f8-4c23-8e2c-9aced50fdc3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:24.050025 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.049973    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60befab0-c2f8-4c23-8e2c-9aced50fdc3d" (UID: "60befab0-c2f8-4c23-8e2c-9aced50fdc3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:24.050124 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.050047    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:24.050124 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.050064    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:24.050124 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.050076    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:24.051864 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.051834    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60befab0-c2f8-4c23-8e2c-9aced50fdc3d" (UID: "60befab0-c2f8-4c23-8e2c-9aced50fdc3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:40:24.051864 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.051854    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-dshm" (OuterVolumeSpecName: "dshm") pod "60befab0-c2f8-4c23-8e2c-9aced50fdc3d" (UID: "60befab0-c2f8-4c23-8e2c-9aced50fdc3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:24.052149 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.052129    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kube-api-access-47frw" (OuterVolumeSpecName: "kube-api-access-47frw") pod "60befab0-c2f8-4c23-8e2c-9aced50fdc3d" (UID: "60befab0-c2f8-4c23-8e2c-9aced50fdc3d"). InnerVolumeSpecName "kube-api-access-47frw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:40:24.105472 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.105418    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "60befab0-c2f8-4c23-8e2c-9aced50fdc3d" (UID: "60befab0-c2f8-4c23-8e2c-9aced50fdc3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:24.151275 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.151241    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:24.151275 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.151275    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47frw\" (UniqueName: \"kubernetes.io/projected/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kube-api-access-47frw\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:24.151275 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.151285    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:24.151536 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.151294    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/60befab0-c2f8-4c23-8e2c-9aced50fdc3d-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:24.463501 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.463402    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:40:24.471371 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.471339    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:40:24.661041 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.661007    2578 generic.go:358] "Generic (PLEG): container finished" podID="60befab0-c2f8-4c23-8e2c-9aced50fdc3d" containerID="4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197" exitCode=0
May 25 11:40:24.661227 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.661090    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"
May 25 11:40:24.661227 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.661086    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w" event={"ID":"60befab0-c2f8-4c23-8e2c-9aced50fdc3d","Type":"ContainerDied","Data":"4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197"}
May 25 11:40:24.661335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.661247    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w" event={"ID":"60befab0-c2f8-4c23-8e2c-9aced50fdc3d","Type":"ContainerDied","Data":"2f15bb49e50907452dfd140004c3fe35a55edfbb268ab67d9559ec322dcb9607"}
May 25 11:40:24.661335 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.661274    2578 scope.go:117] "RemoveContainer" containerID="4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197"
May 25 11:40:24.670792 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.670774    2578 scope.go:117] "RemoveContainer" containerID="83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0"
May 25 11:40:24.681577 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.681348    2578 scope.go:117] "RemoveContainer" containerID="4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197"
May 25 11:40:24.681892 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:40:24.681646    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197\": container with ID starting with 4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197 not found: ID does not exist" containerID="4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197"
May 25 11:40:24.681892 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.681671    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197"} err="failed to get container status \"4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197\": rpc error: code = NotFound desc = could not find container \"4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197\": container with ID starting with 4900f3274ad80634d83adfd705d7e68ca7b8d7c43ee5ff27bde890863c16d197 not found: ID does not exist"
May 25 11:40:24.681892 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.681688    2578 scope.go:117] "RemoveContainer" containerID="83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0"
May 25 11:40:24.682058 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:40:24.681932    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0\": container with ID starting with 83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0 not found: ID does not exist" containerID="83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0"
May 25 11:40:24.682058 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.681958    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0"} err="failed to get container status \"83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0\": rpc error: code = NotFound desc = could not find container \"83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0\": container with ID starting with 83647df80f692e8b4399e5dee6636634e33c224636fcb365a4bf250974a03dc0 not found: ID does not exist"
May 25 11:40:24.688350 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.688323    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"]
May 25 11:40:24.694405 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.694381    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-55c6c598f5-44n6w"]
May 25 11:40:24.934868 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:24.934835    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60befab0-c2f8-4c23-8e2c-9aced50fdc3d" path="/var/lib/kubelet/pods/60befab0-c2f8-4c23-8e2c-9aced50fdc3d/volumes"
May 25 11:40:39.551234 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.551198    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"]
May 25 11:40:39.552152 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.552131    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60befab0-c2f8-4c23-8e2c-9aced50fdc3d" containerName="storage-initializer"
May 25 11:40:39.552255 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.552246    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60befab0-c2f8-4c23-8e2c-9aced50fdc3d" containerName="storage-initializer"
May 25 11:40:39.552338 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.552329    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60befab0-c2f8-4c23-8e2c-9aced50fdc3d" containerName="main"
May 25 11:40:39.552404 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.552396    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60befab0-c2f8-4c23-8e2c-9aced50fdc3d" containerName="main"
May 25 11:40:39.552611 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.552600    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="60befab0-c2f8-4c23-8e2c-9aced50fdc3d" containerName="main"
May 25 11:40:39.570553 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.570507    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.571736 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.571700    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"]
May 25 11:40:39.575995 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.575968    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"conv-test-lora-crit-kserve-self-signed-certs\""
May 25 11:40:39.689020 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.688988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-kserve-provision-location\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.689020 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.689030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-dshm\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.689238 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.689052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc45c874-a35d-4bef-8d4e-966623186721-tls-certs\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.689238 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.689141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-home\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.689238 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.689168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-model-cache\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.689357 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.689261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khcls\" (UniqueName: \"kubernetes.io/projected/dc45c874-a35d-4bef-8d4e-966623186721-kube-api-access-khcls\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.689357 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.689286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-tmp-dir\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.790150 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.790115    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-home\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.790150 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.790153    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-model-cache\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.790401 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.790205    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-khcls\" (UniqueName: \"kubernetes.io/projected/dc45c874-a35d-4bef-8d4e-966623186721-kube-api-access-khcls\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.790401 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.790241    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-tmp-dir\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.790401 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.790276    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-kserve-provision-location\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.790401 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.790304    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-dshm\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.790401 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.790331    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc45c874-a35d-4bef-8d4e-966623186721-tls-certs\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.790694 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.790667    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-home\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.790694 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.790683    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-model-cache\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.790802 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.790733    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-kserve-provision-location\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.790802 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.790758    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-tmp-dir\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.792824 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.792802    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-dshm\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.793056 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.793041    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc45c874-a35d-4bef-8d4e-966623186721-tls-certs\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.805709 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.805645    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khcls\" (UniqueName: \"kubernetes.io/projected/dc45c874-a35d-4bef-8d4e-966623186721-kube-api-access-khcls\") pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") " pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:39.881994 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:39.881956    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:40.031605 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:40.031570    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"]
May 25 11:40:40.031892 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:40:40.031866    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc45c874_a35d_4bef_8d4e_966623186721.slice/crio-713ab29496cc219ebbf2f3d18e32b327fb839252ad2430bb0e8a038b679b79ca WatchSource:0}: Error finding container 713ab29496cc219ebbf2f3d18e32b327fb839252ad2430bb0e8a038b679b79ca: Status 404 returned error can't find the container with id 713ab29496cc219ebbf2f3d18e32b327fb839252ad2430bb0e8a038b679b79ca
May 25 11:40:40.719086 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:40.719055    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-lora-crit-kserve-7b57cd685-2lpt5_dc45c874-a35d-4bef-8d4e-966623186721/storage-initializer/0.log"
May 25 11:40:40.719512 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:40.719102    2578 generic.go:358] "Generic (PLEG): container finished" podID="dc45c874-a35d-4bef-8d4e-966623186721" containerID="6e7f34dcac64393bb0e006154b0e00235f66b5eb62502cb71f38711b5b71d8e4" exitCode=1
May 25 11:40:40.719512 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:40.719193    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5" event={"ID":"dc45c874-a35d-4bef-8d4e-966623186721","Type":"ContainerDied","Data":"6e7f34dcac64393bb0e006154b0e00235f66b5eb62502cb71f38711b5b71d8e4"}
May 25 11:40:40.719512 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:40.719230    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5" event={"ID":"dc45c874-a35d-4bef-8d4e-966623186721","Type":"ContainerStarted","Data":"713ab29496cc219ebbf2f3d18e32b327fb839252ad2430bb0e8a038b679b79ca"}
May 25 11:40:41.724189 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:41.724165    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-lora-crit-kserve-7b57cd685-2lpt5_dc45c874-a35d-4bef-8d4e-966623186721/storage-initializer/1.log"
May 25 11:40:41.724634 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:41.724533    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-lora-crit-kserve-7b57cd685-2lpt5_dc45c874-a35d-4bef-8d4e-966623186721/storage-initializer/0.log"
May 25 11:40:41.724634 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:41.724566    2578 generic.go:358] "Generic (PLEG): container finished" podID="dc45c874-a35d-4bef-8d4e-966623186721" containerID="c80bfda75766ec3539baebccec1d13c017a4163361eb29a69e13237a4505e782" exitCode=1
May 25 11:40:41.724634 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:41.724607    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5" event={"ID":"dc45c874-a35d-4bef-8d4e-966623186721","Type":"ContainerDied","Data":"c80bfda75766ec3539baebccec1d13c017a4163361eb29a69e13237a4505e782"}
May 25 11:40:41.724634 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:41.724633    2578 scope.go:117] "RemoveContainer" containerID="6e7f34dcac64393bb0e006154b0e00235f66b5eb62502cb71f38711b5b71d8e4"
May 25 11:40:41.724967 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:41.724950    2578 scope.go:117] "RemoveContainer" containerID="6e7f34dcac64393bb0e006154b0e00235f66b5eb62502cb71f38711b5b71d8e4"
May 25 11:40:41.735127 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:40:41.735090    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_conv-test-lora-crit-kserve-7b57cd685-2lpt5_kserve-ci-e2e-test_dc45c874-a35d-4bef-8d4e-966623186721_0 in pod sandbox 713ab29496cc219ebbf2f3d18e32b327fb839252ad2430bb0e8a038b679b79ca from index: no such id: '6e7f34dcac64393bb0e006154b0e00235f66b5eb62502cb71f38711b5b71d8e4'" containerID="6e7f34dcac64393bb0e006154b0e00235f66b5eb62502cb71f38711b5b71d8e4"
May 25 11:40:41.735205 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:41.735143    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7f34dcac64393bb0e006154b0e00235f66b5eb62502cb71f38711b5b71d8e4"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_conv-test-lora-crit-kserve-7b57cd685-2lpt5_kserve-ci-e2e-test_dc45c874-a35d-4bef-8d4e-966623186721_0 in pod sandbox 713ab29496cc219ebbf2f3d18e32b327fb839252ad2430bb0e8a038b679b79ca from index: no such id: '6e7f34dcac64393bb0e006154b0e00235f66b5eb62502cb71f38711b5b71d8e4'"
May 25 11:40:41.735325 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:40:41.735306    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=conv-test-lora-crit-kserve-7b57cd685-2lpt5_kserve-ci-e2e-test(dc45c874-a35d-4bef-8d4e-966623186721)\"" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5" podUID="dc45c874-a35d-4bef-8d4e-966623186721"
May 25 11:40:42.736873 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:42.736844    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-lora-crit-kserve-7b57cd685-2lpt5_dc45c874-a35d-4bef-8d4e-966623186721/storage-initializer/1.log"
May 25 11:40:42.737517 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:40:42.737494    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=conv-test-lora-crit-kserve-7b57cd685-2lpt5_kserve-ci-e2e-test(dc45c874-a35d-4bef-8d4e-966623186721)\"" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5" podUID="dc45c874-a35d-4bef-8d4e-966623186721"
May 25 11:40:52.991944 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:52.991911    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"]
May 25 11:40:52.994236 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:52.994219    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:52.999320 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:52.999295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a326b203-cce7-4cad-8491-8349ed2b39b6-tls-certs\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:52.999431 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:52.999341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-home\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:52.999431 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:52.999369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-kserve-provision-location\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:52.999594 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:52.999434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-tmp-dir\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:52.999594 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:52.999479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9sjp\" (UniqueName: \"kubernetes.io/projected/a326b203-cce7-4cad-8491-8349ed2b39b6-kube-api-access-x9sjp\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:52.999594 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:52.999581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-dshm\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:52.999734 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:52.999607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-model-cache\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.000983 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.000956    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 25 11:40:53.040367 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.040334    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"]
May 25 11:40:53.100297 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.100269    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-dshm\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.100497 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.100305    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-model-cache\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.100497 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.100336    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a326b203-cce7-4cad-8491-8349ed2b39b6-tls-certs\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.100497 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.100480    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-home\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.100655 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.100523    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-kserve-provision-location\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.100655 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.100590    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-tmp-dir\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.100655 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.100618    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x9sjp\" (UniqueName: \"kubernetes.io/projected/a326b203-cce7-4cad-8491-8349ed2b39b6-kube-api-access-x9sjp\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.100834 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.100814    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-model-cache\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.100894 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.100856    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-home\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.100984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.100960    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-kserve-provision-location\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.101037 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.100967    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-tmp-dir\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.102807 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.102788    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-dshm\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.103078 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.103060    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a326b203-cce7-4cad-8491-8349ed2b39b6-tls-certs\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.120927 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.120893    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9sjp\" (UniqueName: \"kubernetes.io/projected/a326b203-cce7-4cad-8491-8349ed2b39b6-kube-api-access-x9sjp\") pod \"stop-feature-test-kserve-5579b55ddf-f2v78\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.304478 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.304357    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:40:53.532685 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:40:53.532641    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda326b203_cce7_4cad_8491_8349ed2b39b6.slice/crio-bf314cc43bcfca798a606df83d7cc77b109215240efc9f0d096972e1c0bf4155 WatchSource:0}: Error finding container bf314cc43bcfca798a606df83d7cc77b109215240efc9f0d096972e1c0bf4155: Status 404 returned error can't find the container with id bf314cc43bcfca798a606df83d7cc77b109215240efc9f0d096972e1c0bf4155
May 25 11:40:53.533076 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.533016    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"]
May 25 11:40:53.781329 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.781279    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" event={"ID":"a326b203-cce7-4cad-8491-8349ed2b39b6","Type":"ContainerStarted","Data":"77be5c150793e6ac42b90659dfe40411b4d84ab657554f0cae9bee45cd03184c"}
May 25 11:40:53.781329 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:53.781342    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" event={"ID":"a326b203-cce7-4cad-8491-8349ed2b39b6","Type":"ContainerStarted","Data":"bf314cc43bcfca798a606df83d7cc77b109215240efc9f0d096972e1c0bf4155"}
May 25 11:40:54.788474 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:54.788340    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-lora-crit-kserve-7b57cd685-2lpt5_dc45c874-a35d-4bef-8d4e-966623186721/storage-initializer/2.log"
May 25 11:40:54.789100 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:54.788756    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-lora-crit-kserve-7b57cd685-2lpt5_dc45c874-a35d-4bef-8d4e-966623186721/storage-initializer/1.log"
May 25 11:40:54.789100 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:54.788791    2578 generic.go:358] "Generic (PLEG): container finished" podID="dc45c874-a35d-4bef-8d4e-966623186721" containerID="cf8dde31ed9f48fa84fcc1551eacdb471b5f30806ca22292258dcfc755a95ab9" exitCode=1
May 25 11:40:54.789100 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:54.788891    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5" event={"ID":"dc45c874-a35d-4bef-8d4e-966623186721","Type":"ContainerDied","Data":"cf8dde31ed9f48fa84fcc1551eacdb471b5f30806ca22292258dcfc755a95ab9"}
May 25 11:40:54.789100 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:54.788983    2578 scope.go:117] "RemoveContainer" containerID="c80bfda75766ec3539baebccec1d13c017a4163361eb29a69e13237a4505e782"
May 25 11:40:54.789565 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:54.789536    2578 scope.go:117] "RemoveContainer" containerID="c80bfda75766ec3539baebccec1d13c017a4163361eb29a69e13237a4505e782"
May 25 11:40:54.803339 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:40:54.803303    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_conv-test-lora-crit-kserve-7b57cd685-2lpt5_kserve-ci-e2e-test_dc45c874-a35d-4bef-8d4e-966623186721_1 in pod sandbox 713ab29496cc219ebbf2f3d18e32b327fb839252ad2430bb0e8a038b679b79ca from index: no such id: 'c80bfda75766ec3539baebccec1d13c017a4163361eb29a69e13237a4505e782'" containerID="c80bfda75766ec3539baebccec1d13c017a4163361eb29a69e13237a4505e782"
May 25 11:40:54.803437 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:40:54.803362    2578 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_conv-test-lora-crit-kserve-7b57cd685-2lpt5_kserve-ci-e2e-test_dc45c874-a35d-4bef-8d4e-966623186721_1 in pod sandbox 713ab29496cc219ebbf2f3d18e32b327fb839252ad2430bb0e8a038b679b79ca from index: no such id: 'c80bfda75766ec3539baebccec1d13c017a4163361eb29a69e13237a4505e782'; Skipping pod \"conv-test-lora-crit-kserve-7b57cd685-2lpt5_kserve-ci-e2e-test(dc45c874-a35d-4bef-8d4e-966623186721)\"" logger="UnhandledError"
May 25 11:40:54.805148 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:40:54.804827    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 20s restarting failed container=storage-initializer pod=conv-test-lora-crit-kserve-7b57cd685-2lpt5_kserve-ci-e2e-test(dc45c874-a35d-4bef-8d4e-966623186721)\"" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5" podUID="dc45c874-a35d-4bef-8d4e-966623186721"
May 25 11:40:55.102277 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:55.102176    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"]
May 25 11:40:55.794998 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:55.794966    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-lora-crit-kserve-7b57cd685-2lpt5_dc45c874-a35d-4bef-8d4e-966623186721/storage-initializer/2.log"
May 25 11:40:55.905575 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:55.905540    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"]
May 25 11:40:55.905912 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:55.905883    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main" containerID="cri-o://bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b" gracePeriod=30
May 25 11:40:55.944104 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:55.944081    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-lora-crit-kserve-7b57cd685-2lpt5_dc45c874-a35d-4bef-8d4e-966623186721/storage-initializer/2.log"
May 25 11:40:55.944255 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:55.944144    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:56.024921 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.024877    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-model-cache\") pod \"dc45c874-a35d-4bef-8d4e-966623186721\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") "
May 25 11:40:56.025079 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.024933    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khcls\" (UniqueName: \"kubernetes.io/projected/dc45c874-a35d-4bef-8d4e-966623186721-kube-api-access-khcls\") pod \"dc45c874-a35d-4bef-8d4e-966623186721\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") "
May 25 11:40:56.025079 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.024976    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-home\") pod \"dc45c874-a35d-4bef-8d4e-966623186721\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") "
May 25 11:40:56.025079 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.025027    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-kserve-provision-location\") pod \"dc45c874-a35d-4bef-8d4e-966623186721\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") "
May 25 11:40:56.025235 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.025076    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc45c874-a35d-4bef-8d4e-966623186721-tls-certs\") pod \"dc45c874-a35d-4bef-8d4e-966623186721\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") "
May 25 11:40:56.025235 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.025108    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-dshm\") pod \"dc45c874-a35d-4bef-8d4e-966623186721\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") "
May 25 11:40:56.025488 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.025173    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-tmp-dir\") pod \"dc45c874-a35d-4bef-8d4e-966623186721\" (UID: \"dc45c874-a35d-4bef-8d4e-966623186721\") "
May 25 11:40:56.025605 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.025164    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc45c874-a35d-4bef-8d4e-966623186721" (UID: "dc45c874-a35d-4bef-8d4e-966623186721"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:56.025605 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.025323    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-home" (OuterVolumeSpecName: "home") pod "dc45c874-a35d-4bef-8d4e-966623186721" (UID: "dc45c874-a35d-4bef-8d4e-966623186721"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:56.025605 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.025431    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc45c874-a35d-4bef-8d4e-966623186721" (UID: "dc45c874-a35d-4bef-8d4e-966623186721"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:56.025605 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.025494    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dc45c874-a35d-4bef-8d4e-966623186721" (UID: "dc45c874-a35d-4bef-8d4e-966623186721"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:56.026144 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.025817    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:56.026144 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.025847    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:56.026144 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.026013    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:56.026144 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.026034    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:56.027362 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.027335    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc45c874-a35d-4bef-8d4e-966623186721-kube-api-access-khcls" (OuterVolumeSpecName: "kube-api-access-khcls") pod "dc45c874-a35d-4bef-8d4e-966623186721" (UID: "dc45c874-a35d-4bef-8d4e-966623186721"). InnerVolumeSpecName "kube-api-access-khcls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:40:56.027521 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.027497    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-dshm" (OuterVolumeSpecName: "dshm") pod "dc45c874-a35d-4bef-8d4e-966623186721" (UID: "dc45c874-a35d-4bef-8d4e-966623186721"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:40:56.027579 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.027561    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc45c874-a35d-4bef-8d4e-966623186721-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc45c874-a35d-4bef-8d4e-966623186721" (UID: "dc45c874-a35d-4bef-8d4e-966623186721"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:40:56.126833 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.126744    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khcls\" (UniqueName: \"kubernetes.io/projected/dc45c874-a35d-4bef-8d4e-966623186721-kube-api-access-khcls\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:56.126833 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.126778    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc45c874-a35d-4bef-8d4e-966623186721-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:56.126833 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.126792    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc45c874-a35d-4bef-8d4e-966623186721-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:40:56.800555 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.800524    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_conv-test-lora-crit-kserve-7b57cd685-2lpt5_dc45c874-a35d-4bef-8d4e-966623186721/storage-initializer/2.log"
May 25 11:40:56.800929 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.800641    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"
May 25 11:40:56.800929 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.800691    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5" event={"ID":"dc45c874-a35d-4bef-8d4e-966623186721","Type":"ContainerDied","Data":"713ab29496cc219ebbf2f3d18e32b327fb839252ad2430bb0e8a038b679b79ca"}
May 25 11:40:56.800929 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.800762    2578 scope.go:117] "RemoveContainer" containerID="cf8dde31ed9f48fa84fcc1551eacdb471b5f30806ca22292258dcfc755a95ab9"
May 25 11:40:56.851517 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.851475    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"]
May 25 11:40:56.855634 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.855606    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/conv-test-lora-crit-kserve-7b57cd685-2lpt5"]
May 25 11:40:56.935656 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:56.935617    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc45c874-a35d-4bef-8d4e-966623186721" path="/var/lib/kubelet/pods/dc45c874-a35d-4bef-8d4e-966623186721/volumes"
May 25 11:40:57.807304 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:57.807212    2578 generic.go:358] "Generic (PLEG): container finished" podID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerID="77be5c150793e6ac42b90659dfe40411b4d84ab657554f0cae9bee45cd03184c" exitCode=0
May 25 11:40:57.807304 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:57.807265    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" event={"ID":"a326b203-cce7-4cad-8491-8349ed2b39b6","Type":"ContainerDied","Data":"77be5c150793e6ac42b90659dfe40411b4d84ab657554f0cae9bee45cd03184c"}
May 25 11:40:58.813864 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:58.813824    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" event={"ID":"a326b203-cce7-4cad-8491-8349ed2b39b6","Type":"ContainerStarted","Data":"cf4e4f8576103d1662794eba8cc08d3f8407f0454051b80a293c54b6a93f2ab8"}
May 25 11:40:58.843012 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:40:58.842958    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" podStartSLOduration=6.842942317 podStartE2EDuration="6.842942317s" podCreationTimestamp="2026-05-25 11:40:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:40:58.840604525 +0000 UTC m=+1118.489889521" watchObservedRunningTime="2026-05-25 11:40:58.842942317 +0000 UTC m=+1118.492227312"
May 25 11:41:03.304695 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:03.304661    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:41:03.304695 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:03.304706    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:41:03.306398 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:03.306368    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 11:41:13.305516 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:13.305427    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 11:41:23.305753 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:23.305700    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 11:41:26.148032 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.148008    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7_1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6/main/0.log"
May 25 11:41:26.148488 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.148443    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:41:26.284735 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.284691    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-home\") pod \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") "
May 25 11:41:26.284936 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.284764    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kserve-provision-location\") pod \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") "
May 25 11:41:26.284936 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.284815    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-model-cache\") pod \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") "
May 25 11:41:26.284936 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.284852    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tmp-dir\") pod \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") "
May 25 11:41:26.284936 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.284895    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcwks\" (UniqueName: \"kubernetes.io/projected/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kube-api-access-vcwks\") pod \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") "
May 25 11:41:26.284936 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.284928    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tls-certs\") pod \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") "
May 25 11:41:26.285211 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.284956    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-dshm\") pod \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\" (UID: \"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6\") "
May 25 11:41:26.285211 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.285112    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" (UID: "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:41:26.285310 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.285254    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:41:26.285367 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.285332    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-home" (OuterVolumeSpecName: "home") pod "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" (UID: "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:41:26.287360 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.287323    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kube-api-access-vcwks" (OuterVolumeSpecName: "kube-api-access-vcwks") pod "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" (UID: "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6"). InnerVolumeSpecName "kube-api-access-vcwks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:41:26.287892 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.287859    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-dshm" (OuterVolumeSpecName: "dshm") pod "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" (UID: "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:41:26.288010 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.287960    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" (UID: "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:41:26.303065 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.303034    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" (UID: "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:41:26.347764 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.347714    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" (UID: "1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:41:26.385667 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.385629    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:41:26.385667 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.385662    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:41:26.385667 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.385674    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:41:26.385896 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.385683    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcwks\" (UniqueName: \"kubernetes.io/projected/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-kube-api-access-vcwks\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:41:26.385896 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.385693    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:41:26.385896 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.385701    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:41:26.922398 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.922349    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7_1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6/main/0.log"
May 25 11:41:26.922900 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.922865    2578 generic.go:358] "Generic (PLEG): container finished" podID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerID="bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b" exitCode=137
May 25 11:41:26.923027 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.923002    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" event={"ID":"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6","Type":"ContainerDied","Data":"bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b"}
May 25 11:41:26.923093 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.923047    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7" event={"ID":"1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6","Type":"ContainerDied","Data":"8b2072f5f1c9f7e8c49b3f3ab0e0ae65f7ac05aa0d6a065438d4494953cd6131"}
May 25 11:41:26.923093 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.923072    2578 scope.go:117] "RemoveContainer" containerID="bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b"
May 25 11:41:26.923317 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.923299    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"
May 25 11:41:26.944224 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.944204    2578 scope.go:117] "RemoveContainer" containerID="1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1"
May 25 11:41:26.959004 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.958974    2578 scope.go:117] "RemoveContainer" containerID="bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b"
May 25 11:41:26.959389 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:41:26.959368    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b\": container with ID starting with bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b not found: ID does not exist" containerID="bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b"
May 25 11:41:26.959521 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.959400    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b"} err="failed to get container status \"bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b\": rpc error: code = NotFound desc = could not find container \"bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b\": container with ID starting with bb725f6db359c84e617179aff2ab7cbd7b3b7f4f1dfe39826614ca771545ff7b not found: ID does not exist"
May 25 11:41:26.959521 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.959420    2578 scope.go:117] "RemoveContainer" containerID="1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1"
May 25 11:41:26.960039 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:41:26.960016    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1\": container with ID starting with 1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1 not found: ID does not exist" containerID="1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1"
May 25 11:41:26.960108 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.960045    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1"} err="failed to get container status \"1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1\": rpc error: code = NotFound desc = could not find container \"1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1\": container with ID starting with 1d028d82f574d47f829a6d4a3ed903e6c4cab4dc2785de6d1ec77b77c1d2a1a1 not found: ID does not exist"
May 25 11:41:26.966882 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.966858    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"]
May 25 11:41:26.974889 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:26.974866    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-69dd5548699fdr7"]
May 25 11:41:28.934877 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:28.934833    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" path="/var/lib/kubelet/pods/1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6/volumes"
May 25 11:41:29.518191 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518162    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"]
May 25 11:41:29.518576 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518547    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc45c874-a35d-4bef-8d4e-966623186721" containerName="storage-initializer"
May 25 11:41:29.518576 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518579    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc45c874-a35d-4bef-8d4e-966623186721" containerName="storage-initializer"
May 25 11:41:29.518730 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518592    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="storage-initializer"
May 25 11:41:29.518730 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518600    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="storage-initializer"
May 25 11:41:29.518730 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518614    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc45c874-a35d-4bef-8d4e-966623186721" containerName="storage-initializer"
May 25 11:41:29.518730 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518622    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc45c874-a35d-4bef-8d4e-966623186721" containerName="storage-initializer"
May 25 11:41:29.518730 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518644    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main"
May 25 11:41:29.518730 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518655    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main"
May 25 11:41:29.518730 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518729    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc45c874-a35d-4bef-8d4e-966623186721" containerName="storage-initializer"
May 25 11:41:29.519058 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518744    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="1b4ab078-14d3-4c70-ab2a-eaf6bb4439c6" containerName="main"
May 25 11:41:29.519058 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518758    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc45c874-a35d-4bef-8d4e-966623186721" containerName="storage-initializer"
May 25 11:41:29.519058 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518844    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc45c874-a35d-4bef-8d4e-966623186721" containerName="storage-initializer"
May 25 11:41:29.519058 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518854    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc45c874-a35d-4bef-8d4e-966623186721" containerName="storage-initializer"
May 25 11:41:29.519058 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.518928    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc45c874-a35d-4bef-8d4e-966623186721" containerName="storage-initializer"
May 25 11:41:29.522700 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.522680    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.526447 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.526424    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 25 11:41:29.545380 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.545351    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"]
May 25 11:41:29.611674 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.611622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-model-cache\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.611870 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.611708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-dshm\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.611870 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.611750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-tmp-dir\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.611870 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.611817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6691d89-304c-495d-8cd5-cf31f0f4c679-tls-certs\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.611870 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.611846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rnn\" (UniqueName: \"kubernetes.io/projected/c6691d89-304c-495d-8cd5-cf31f0f4c679-kube-api-access-g8rnn\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.612077 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.611896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-home\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.612077 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.611985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.712913 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.712865    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-home\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.713079 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.712963    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.713079 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.713013    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-model-cache\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.713079 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.713071    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-dshm\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.713228 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.713101    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-tmp-dir\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.713228 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.713174    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6691d89-304c-495d-8cd5-cf31f0f4c679-tls-certs\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.713228 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.713204    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-home\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.713228 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.713222    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rnn\" (UniqueName: \"kubernetes.io/projected/c6691d89-304c-495d-8cd5-cf31f0f4c679-kube-api-access-g8rnn\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.713411 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.713368    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.713497 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.713446    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-tmp-dir\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.713586 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.713566    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-model-cache\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.715544 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.715518    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-dshm\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.715768 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.715748    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6691d89-304c-495d-8cd5-cf31f0f4c679-tls-certs\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.735394 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.735360    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rnn\" (UniqueName: \"kubernetes.io/projected/c6691d89-304c-495d-8cd5-cf31f0f4c679-kube-api-access-g8rnn\") pod \"custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:29.833108 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:29.832994    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:41:30.036320 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:30.036291    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"]
May 25 11:41:30.038797 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:41:30.038768    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6691d89_304c_495d_8cd5_cf31f0f4c679.slice/crio-9c29166169018eccd194dae8fde746a83c82006c0c965c81b06f37c438c35e29 WatchSource:0}: Error finding container 9c29166169018eccd194dae8fde746a83c82006c0c965c81b06f37c438c35e29: Status 404 returned error can't find the container with id 9c29166169018eccd194dae8fde746a83c82006c0c965c81b06f37c438c35e29
May 25 11:41:30.943532 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:30.943490    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" event={"ID":"c6691d89-304c-495d-8cd5-cf31f0f4c679","Type":"ContainerStarted","Data":"0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b"}
May 25 11:41:30.943721 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:30.943538    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" event={"ID":"c6691d89-304c-495d-8cd5-cf31f0f4c679","Type":"ContainerStarted","Data":"9c29166169018eccd194dae8fde746a83c82006c0c965c81b06f37c438c35e29"}
May 25 11:41:33.305597 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:33.305542    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 11:41:43.305755 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:43.305645    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 11:41:53.305513 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:41:53.305436    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 11:42:03.305561 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:03.305513    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 11:42:13.305278 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:13.305228    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 11:42:20.952877 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:20.952849    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:42:20.957516 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:20.957492    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:42:23.305766 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:23.305710    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
May 25 11:42:33.183986 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:33.183953    2578 generic.go:358] "Generic (PLEG): container finished" podID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerID="0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b" exitCode=0
May 25 11:42:33.184433 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:33.184021    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" event={"ID":"c6691d89-304c-495d-8cd5-cf31f0f4c679","Type":"ContainerDied","Data":"0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b"}
May 25 11:42:33.315477 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:33.315433    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:42:33.323556 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:33.323525    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:42:34.196207 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:34.196172    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" event={"ID":"c6691d89-304c-495d-8cd5-cf31f0f4c679","Type":"ContainerStarted","Data":"2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9"}
May 25 11:42:34.233982 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:34.233931    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" podStartSLOduration=65.233916753 podStartE2EDuration="1m5.233916753s" podCreationTimestamp="2026-05-25 11:41:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:42:34.233430864 +0000 UTC m=+1213.882715860" watchObservedRunningTime="2026-05-25 11:42:34.233916753 +0000 UTC m=+1213.883201747"
May 25 11:42:35.163344 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:35.163300    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"]
May 25 11:42:35.200267 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:35.200224    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main" containerID="cri-o://cf4e4f8576103d1662794eba8cc08d3f8407f0454051b80a293c54b6a93f2ab8" gracePeriod=30
May 25 11:42:39.833860 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:39.833818    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:42:39.834322 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:39.833878    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:42:39.835270 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:39.835240    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 11:42:49.833644 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:49.833599    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 11:42:57.235513 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.235478    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"]
May 25 11:42:57.239331 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.239309    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.249580 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.249552    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"]
May 25 11:42:57.368096 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.368053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-home\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.368292 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.368122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d36a0111-eef5-423c-b113-22929de22f9a-tls-certs\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.368292 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.368151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-tmp-dir\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.368292 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.368178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-model-cache\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.368438 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.368274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-kserve-provision-location\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.368438 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.368319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2brk\" (UniqueName: \"kubernetes.io/projected/d36a0111-eef5-423c-b113-22929de22f9a-kube-api-access-c2brk\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.368438 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.368357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-dshm\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.469242 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.469201    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d36a0111-eef5-423c-b113-22929de22f9a-tls-certs\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.469487 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.469257    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-tmp-dir\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.469487 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.469292    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-model-cache\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.469487 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.469333    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-kserve-provision-location\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.469487 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.469360    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c2brk\" (UniqueName: \"kubernetes.io/projected/d36a0111-eef5-423c-b113-22929de22f9a-kube-api-access-c2brk\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.469487 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.469394    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-dshm\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.469946 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.469762    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-model-cache\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.469946 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.469784    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-tmp-dir\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.469946 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.469793    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-home\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.470260 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.470052    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-kserve-provision-location\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.470260 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.470073    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-home\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.471925 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.471903    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-dshm\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.472084 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.472067    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d36a0111-eef5-423c-b113-22929de22f9a-tls-certs\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.488843 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.488769    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2brk\" (UniqueName: \"kubernetes.io/projected/d36a0111-eef5-423c-b113-22929de22f9a-kube-api-access-c2brk\") pod \"stop-feature-test-kserve-5579b55ddf-b726r\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.550675 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.550626    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:42:57.703411 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:57.703375    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"]
May 25 11:42:57.703546 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:42:57.703415    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36a0111_eef5_423c_b113_22929de22f9a.slice/crio-aece6ea616c3b77e94f2f2e9d7ff930d3b1072c4074f00700b69c156eb90248a WatchSource:0}: Error finding container aece6ea616c3b77e94f2f2e9d7ff930d3b1072c4074f00700b69c156eb90248a: Status 404 returned error can't find the container with id aece6ea616c3b77e94f2f2e9d7ff930d3b1072c4074f00700b69c156eb90248a
May 25 11:42:58.293375 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:58.293333    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" event={"ID":"d36a0111-eef5-423c-b113-22929de22f9a","Type":"ContainerStarted","Data":"ffc60fd3eebdfad8904dbca9df030a730a5827dbd16bc6ca0799166d4d29e1d1"}
May 25 11:42:58.293788 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:58.293379    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" event={"ID":"d36a0111-eef5-423c-b113-22929de22f9a","Type":"ContainerStarted","Data":"aece6ea616c3b77e94f2f2e9d7ff930d3b1072c4074f00700b69c156eb90248a"}
May 25 11:42:59.833792 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:42:59.833674    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 11:43:05.328495 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.328440    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5579b55ddf-f2v78_a326b203-cce7-4cad-8491-8349ed2b39b6/main/0.log"
May 25 11:43:05.328952 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.328825    2578 generic.go:358] "Generic (PLEG): container finished" podID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerID="cf4e4f8576103d1662794eba8cc08d3f8407f0454051b80a293c54b6a93f2ab8" exitCode=137
May 25 11:43:05.328952 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.328882    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" event={"ID":"a326b203-cce7-4cad-8491-8349ed2b39b6","Type":"ContainerDied","Data":"cf4e4f8576103d1662794eba8cc08d3f8407f0454051b80a293c54b6a93f2ab8"}
May 25 11:43:05.431827 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.431803    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5579b55ddf-f2v78_a326b203-cce7-4cad-8491-8349ed2b39b6/main/0.log"
May 25 11:43:05.432200 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.432180    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:43:05.543489 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.543444    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-dshm\") pod \"a326b203-cce7-4cad-8491-8349ed2b39b6\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") "
May 25 11:43:05.543679 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.543509    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-kserve-provision-location\") pod \"a326b203-cce7-4cad-8491-8349ed2b39b6\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") "
May 25 11:43:05.543679 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.543546    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-model-cache\") pod \"a326b203-cce7-4cad-8491-8349ed2b39b6\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") "
May 25 11:43:05.543679 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.543573    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-home\") pod \"a326b203-cce7-4cad-8491-8349ed2b39b6\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") "
May 25 11:43:05.543679 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.543639    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-tmp-dir\") pod \"a326b203-cce7-4cad-8491-8349ed2b39b6\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") "
May 25 11:43:05.543679 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.543675    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a326b203-cce7-4cad-8491-8349ed2b39b6-tls-certs\") pod \"a326b203-cce7-4cad-8491-8349ed2b39b6\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") "
May 25 11:43:05.543957 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.543702    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9sjp\" (UniqueName: \"kubernetes.io/projected/a326b203-cce7-4cad-8491-8349ed2b39b6-kube-api-access-x9sjp\") pod \"a326b203-cce7-4cad-8491-8349ed2b39b6\" (UID: \"a326b203-cce7-4cad-8491-8349ed2b39b6\") "
May 25 11:43:05.543957 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.543846    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "a326b203-cce7-4cad-8491-8349ed2b39b6" (UID: "a326b203-cce7-4cad-8491-8349ed2b39b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:05.544069 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.543969    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:43:05.544484 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.544420    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-home" (OuterVolumeSpecName: "home") pod "a326b203-cce7-4cad-8491-8349ed2b39b6" (UID: "a326b203-cce7-4cad-8491-8349ed2b39b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:05.545984 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.545959    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-dshm" (OuterVolumeSpecName: "dshm") pod "a326b203-cce7-4cad-8491-8349ed2b39b6" (UID: "a326b203-cce7-4cad-8491-8349ed2b39b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:05.546204 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.546183    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a326b203-cce7-4cad-8491-8349ed2b39b6-kube-api-access-x9sjp" (OuterVolumeSpecName: "kube-api-access-x9sjp") pod "a326b203-cce7-4cad-8491-8349ed2b39b6" (UID: "a326b203-cce7-4cad-8491-8349ed2b39b6"). InnerVolumeSpecName "kube-api-access-x9sjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:43:05.546491 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.546452    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a326b203-cce7-4cad-8491-8349ed2b39b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a326b203-cce7-4cad-8491-8349ed2b39b6" (UID: "a326b203-cce7-4cad-8491-8349ed2b39b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:43:05.557129 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.557090    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a326b203-cce7-4cad-8491-8349ed2b39b6" (UID: "a326b203-cce7-4cad-8491-8349ed2b39b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:05.614203 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.614128    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a326b203-cce7-4cad-8491-8349ed2b39b6" (UID: "a326b203-cce7-4cad-8491-8349ed2b39b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:43:05.644952 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.644917    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:43:05.644952 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.644948    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:43:05.644952 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.644959    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:43:05.645257 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.644969    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a326b203-cce7-4cad-8491-8349ed2b39b6-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:43:05.645257 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.644978    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a326b203-cce7-4cad-8491-8349ed2b39b6-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:43:05.645257 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:05.644986    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9sjp\" (UniqueName: \"kubernetes.io/projected/a326b203-cce7-4cad-8491-8349ed2b39b6-kube-api-access-x9sjp\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:43:06.334189 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:06.334156    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5579b55ddf-f2v78_a326b203-cce7-4cad-8491-8349ed2b39b6/main/0.log"
May 25 11:43:06.334721 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:06.334530    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78" event={"ID":"a326b203-cce7-4cad-8491-8349ed2b39b6","Type":"ContainerDied","Data":"bf314cc43bcfca798a606df83d7cc77b109215240efc9f0d096972e1c0bf4155"}
May 25 11:43:06.334721 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:06.334569    2578 scope.go:117] "RemoveContainer" containerID="cf4e4f8576103d1662794eba8cc08d3f8407f0454051b80a293c54b6a93f2ab8"
May 25 11:43:06.334721 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:06.334595    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"
May 25 11:43:06.348231 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:06.348209    2578 scope.go:117] "RemoveContainer" containerID="77be5c150793e6ac42b90659dfe40411b4d84ab657554f0cae9bee45cd03184c"
May 25 11:43:06.361861 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:06.361808    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"]
May 25 11:43:06.364780 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:06.364754    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-f2v78"]
May 25 11:43:06.941643 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:06.941608    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" path="/var/lib/kubelet/pods/a326b203-cce7-4cad-8491-8349ed2b39b6/volumes"
May 25 11:43:09.833846 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:09.833754    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 11:43:19.833750 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:19.833704    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 11:43:29.834080 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:29.834022    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 11:43:39.834171 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:39.834117    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 11:43:49.833512 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:49.833428    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 11:43:59.834129 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:43:59.834025    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 25 11:44:09.588402 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:09.588364    2578 generic.go:358] "Generic (PLEG): container finished" podID="d36a0111-eef5-423c-b113-22929de22f9a" containerID="ffc60fd3eebdfad8904dbca9df030a730a5827dbd16bc6ca0799166d4d29e1d1" exitCode=0
May 25 11:44:09.588898 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:09.588445    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" event={"ID":"d36a0111-eef5-423c-b113-22929de22f9a","Type":"ContainerDied","Data":"ffc60fd3eebdfad8904dbca9df030a730a5827dbd16bc6ca0799166d4d29e1d1"}
May 25 11:44:09.589744 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:09.589729    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:44:09.843808 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:09.843776    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:44:09.852415 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:09.852371    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:44:10.594649 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:10.594608    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" event={"ID":"d36a0111-eef5-423c-b113-22929de22f9a","Type":"ContainerStarted","Data":"cb67437902f70b98b334d84ae0a380caf9b9a8131f883056444a9a9bd18b6be9"}
May 25 11:44:10.618923 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:10.618870    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" podStartSLOduration=73.618853653 podStartE2EDuration="1m13.618853653s" podCreationTimestamp="2026-05-25 11:42:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:44:10.616844404 +0000 UTC m=+1310.266129399" watchObservedRunningTime="2026-05-25 11:44:10.618853653 +0000 UTC m=+1310.268138698"
May 25 11:44:15.973534 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:15.973494    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"]
May 25 11:44:15.974087 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:15.973871    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main" containerID="cri-o://2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9" gracePeriod=30
May 25 11:44:17.551441 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:17.551387    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:44:17.551441 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:17.551444    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:44:17.552989 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:17.552962    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 25 11:44:27.551176 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:27.551121    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 25 11:44:31.044945 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.044910    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"]
May 25 11:44:31.045403 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.045358    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main"
May 25 11:44:31.045403 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.045377    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main"
May 25 11:44:31.045403 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.045395    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="storage-initializer"
May 25 11:44:31.045403 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.045406    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="storage-initializer"
May 25 11:44:31.045659 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.045524    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a326b203-cce7-4cad-8491-8349ed2b39b6" containerName="main"
May 25 11:44:31.047977 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.047957    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.057234 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.057207    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 25 11:44:31.070408 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.070381    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"]
May 25 11:44:31.082112 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.082081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/677e8234-983d-43c6-a498-56ae63d43f2c-tls-certs\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.082313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.082131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhsn\" (UniqueName: \"kubernetes.io/projected/677e8234-983d-43c6-a498-56ae63d43f2c-kube-api-access-2dhsn\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.082313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.082153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-tmp-dir\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.082313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.082181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-home\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.082313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.082218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-model-cache\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.082313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.082242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-kserve-provision-location\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.082313 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.082267    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-dshm\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.183695 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.183662    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-kserve-provision-location\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.183695 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.183703    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-dshm\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.183932 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.183755    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/677e8234-983d-43c6-a498-56ae63d43f2c-tls-certs\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.183932 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.183808    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhsn\" (UniqueName: \"kubernetes.io/projected/677e8234-983d-43c6-a498-56ae63d43f2c-kube-api-access-2dhsn\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.183932 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.183835    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-tmp-dir\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.183932 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.183859    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-home\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.183932 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.183902    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-model-cache\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.184142 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.184096    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-kserve-provision-location\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.184378 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.184346    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-tmp-dir\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.184536 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.184392    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-model-cache\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.184536 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.184441    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-home\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.186324 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.186295    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-dshm\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.186603 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.186584    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/677e8234-983d-43c6-a498-56ae63d43f2c-tls-certs\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.194694 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.194664    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhsn\" (UniqueName: \"kubernetes.io/projected/677e8234-983d-43c6-a498-56ae63d43f2c-kube-api-access-2dhsn\") pod \"router-with-refs-test-kserve-7f9485d6b5-jw4rk\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.358385 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.358285    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:44:31.499651 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.499615    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"]
May 25 11:44:31.503162 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:44:31.503121    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677e8234_983d_43c6_a498_56ae63d43f2c.slice/crio-bb7f41c9555f0b3e10a18e02efda3bb99ed9e101b3a992baa8e1712e3bad5b53 WatchSource:0}: Error finding container bb7f41c9555f0b3e10a18e02efda3bb99ed9e101b3a992baa8e1712e3bad5b53: Status 404 returned error can't find the container with id bb7f41c9555f0b3e10a18e02efda3bb99ed9e101b3a992baa8e1712e3bad5b53
May 25 11:44:31.689265 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.689175    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk" event={"ID":"677e8234-983d-43c6-a498-56ae63d43f2c","Type":"ContainerStarted","Data":"b414534c4400ea0dbb7041e5ee78c3bcd8f065eb2bcff94d47806aa3101e8c2a"}
May 25 11:44:31.689265 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:31.689219    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk" event={"ID":"677e8234-983d-43c6-a498-56ae63d43f2c","Type":"ContainerStarted","Data":"bb7f41c9555f0b3e10a18e02efda3bb99ed9e101b3a992baa8e1712e3bad5b53"}
May 25 11:44:37.551115 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:37.551062    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 25 11:44:46.224824 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.224791    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6_c6691d89-304c-495d-8cd5-cf31f0f4c679/main/0.log"
May 25 11:44:46.225233 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.225196    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:44:46.322032 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.321941    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-dshm\") pod \"c6691d89-304c-495d-8cd5-cf31f0f4c679\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") "
May 25 11:44:46.322032 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.322014    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-tmp-dir\") pod \"c6691d89-304c-495d-8cd5-cf31f0f4c679\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") "
May 25 11:44:46.322291 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.322058    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-home\") pod \"c6691d89-304c-495d-8cd5-cf31f0f4c679\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") "
May 25 11:44:46.322291 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.322107    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-kserve-provision-location\") pod \"c6691d89-304c-495d-8cd5-cf31f0f4c679\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") "
May 25 11:44:46.322291 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.322155    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8rnn\" (UniqueName: \"kubernetes.io/projected/c6691d89-304c-495d-8cd5-cf31f0f4c679-kube-api-access-g8rnn\") pod \"c6691d89-304c-495d-8cd5-cf31f0f4c679\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") "
May 25 11:44:46.322291 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.322181    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-model-cache\") pod \"c6691d89-304c-495d-8cd5-cf31f0f4c679\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") "
May 25 11:44:46.322291 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.322217    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6691d89-304c-495d-8cd5-cf31f0f4c679-tls-certs\") pod \"c6691d89-304c-495d-8cd5-cf31f0f4c679\" (UID: \"c6691d89-304c-495d-8cd5-cf31f0f4c679\") "
May 25 11:44:46.325196 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.324738    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-dshm" (OuterVolumeSpecName: "dshm") pod "c6691d89-304c-495d-8cd5-cf31f0f4c679" (UID: "c6691d89-304c-495d-8cd5-cf31f0f4c679"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:44:46.325449 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.325411    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-home" (OuterVolumeSpecName: "home") pod "c6691d89-304c-495d-8cd5-cf31f0f4c679" (UID: "c6691d89-304c-495d-8cd5-cf31f0f4c679"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:44:46.328633 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.328595    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6691d89-304c-495d-8cd5-cf31f0f4c679" (UID: "c6691d89-304c-495d-8cd5-cf31f0f4c679"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:44:46.328900 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.328871    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6691d89-304c-495d-8cd5-cf31f0f4c679-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6691d89-304c-495d-8cd5-cf31f0f4c679" (UID: "c6691d89-304c-495d-8cd5-cf31f0f4c679"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:44:46.330834 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.330805    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6691d89-304c-495d-8cd5-cf31f0f4c679-kube-api-access-g8rnn" (OuterVolumeSpecName: "kube-api-access-g8rnn") pod "c6691d89-304c-495d-8cd5-cf31f0f4c679" (UID: "c6691d89-304c-495d-8cd5-cf31f0f4c679"). InnerVolumeSpecName "kube-api-access-g8rnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:44:46.338504 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.338448    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6691d89-304c-495d-8cd5-cf31f0f4c679" (UID: "c6691d89-304c-495d-8cd5-cf31f0f4c679"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:44:46.386925 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.386876    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6691d89-304c-495d-8cd5-cf31f0f4c679" (UID: "c6691d89-304c-495d-8cd5-cf31f0f4c679"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:44:46.423638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.423598    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:44:46.423638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.423629    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:44:46.423638 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.423639    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:44:46.423869 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.423650    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8rnn\" (UniqueName: \"kubernetes.io/projected/c6691d89-304c-495d-8cd5-cf31f0f4c679-kube-api-access-g8rnn\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:44:46.423869 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.423660    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:44:46.423869 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.423669    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6691d89-304c-495d-8cd5-cf31f0f4c679-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:44:46.423869 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.423676    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6691d89-304c-495d-8cd5-cf31f0f4c679-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:44:46.746161 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.746132    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6_c6691d89-304c-495d-8cd5-cf31f0f4c679/main/0.log"
May 25 11:44:46.746497 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.746448    2578 generic.go:358] "Generic (PLEG): container finished" podID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerID="2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9" exitCode=137
May 25 11:44:46.746592 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.746510    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" event={"ID":"c6691d89-304c-495d-8cd5-cf31f0f4c679","Type":"ContainerDied","Data":"2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9"}
May 25 11:44:46.746592 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.746542    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6" event={"ID":"c6691d89-304c-495d-8cd5-cf31f0f4c679","Type":"ContainerDied","Data":"9c29166169018eccd194dae8fde746a83c82006c0c965c81b06f37c438c35e29"}
May 25 11:44:46.746592 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.746555    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"
May 25 11:44:46.746761 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.746561    2578 scope.go:117] "RemoveContainer" containerID="2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9"
May 25 11:44:46.757599 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.757577    2578 scope.go:117] "RemoveContainer" containerID="0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b"
May 25 11:44:46.768953 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.768924    2578 scope.go:117] "RemoveContainer" containerID="2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9"
May 25 11:44:46.769350 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:44:46.769321    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9\": container with ID starting with 2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9 not found: ID does not exist" containerID="2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9"
May 25 11:44:46.769517 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.769359    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9"} err="failed to get container status \"2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9\": rpc error: code = NotFound desc = could not find container \"2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9\": container with ID starting with 2599f7e0265500235fbe76263f99dbb23e988a53dacba0e1f2be216c0a6deff9 not found: ID does not exist"
May 25 11:44:46.769517 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.769384    2578 scope.go:117] "RemoveContainer" containerID="0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b"
May 25 11:44:46.769702 ip-10-0-133-58 kubenswrapper[2578]: E0525 11:44:46.769686    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b\": container with ID starting with 0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b not found: ID does not exist" containerID="0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b"
May 25 11:44:46.769762 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.769709    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b"} err="failed to get container status \"0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b\": rpc error: code = NotFound desc = could not find container \"0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b\": container with ID starting with 0e561f42dc553707776b00040f261a8f0c1c5b9c4fe7c3d8391c487f08e6c86b not found: ID does not exist"
May 25 11:44:46.771079 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.771055    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"]
May 25 11:44:46.775468 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.775426    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77dc8d7f7c-hp8v6"]
May 25 11:44:46.934783 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:46.934746    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" path="/var/lib/kubelet/pods/c6691d89-304c-495d-8cd5-cf31f0f4c679/volumes"
May 25 11:44:47.551590 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:47.551548    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 25 11:44:57.552144 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:44:57.552090    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 25 11:45:07.551141 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:45:07.551089    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 25 11:45:17.551578 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:45:17.551535    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 25 11:45:27.551429 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:45:27.551373    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 25 11:45:37.551400 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:45:37.551349    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 25 11:45:47.561981 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:45:47.561948    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:45:47.569944 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:45:47.569919    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:45:48.730249 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:45:48.730216    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"]
May 25 11:45:49.004103 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:45:49.004007    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main" containerID="cri-o://cb67437902f70b98b334d84ae0a380caf9b9a8131f883056444a9a9bd18b6be9" gracePeriod=30
May 25 11:46:19.115268 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.115241    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5579b55ddf-b726r_d36a0111-eef5-423c-b113-22929de22f9a/main/0.log"
May 25 11:46:19.115729 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.115603    2578 generic.go:358] "Generic (PLEG): container finished" podID="d36a0111-eef5-423c-b113-22929de22f9a" containerID="cb67437902f70b98b334d84ae0a380caf9b9a8131f883056444a9a9bd18b6be9" exitCode=137
May 25 11:46:19.115729 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.115673    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" event={"ID":"d36a0111-eef5-423c-b113-22929de22f9a","Type":"ContainerDied","Data":"cb67437902f70b98b334d84ae0a380caf9b9a8131f883056444a9a9bd18b6be9"}
May 25 11:46:19.233169 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.233142    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5579b55ddf-b726r_d36a0111-eef5-423c-b113-22929de22f9a/main/0.log"
May 25 11:46:19.233533 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.233516    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:46:19.289884 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.289800    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-tmp-dir\") pod \"d36a0111-eef5-423c-b113-22929de22f9a\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") "
May 25 11:46:19.289884 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.289839    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-dshm\") pod \"d36a0111-eef5-423c-b113-22929de22f9a\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") "
May 25 11:46:19.290107 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.289893    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-model-cache\") pod \"d36a0111-eef5-423c-b113-22929de22f9a\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") "
May 25 11:46:19.290107 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.289961    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-home\") pod \"d36a0111-eef5-423c-b113-22929de22f9a\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") "
May 25 11:46:19.290107 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.289988    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-kserve-provision-location\") pod \"d36a0111-eef5-423c-b113-22929de22f9a\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") "
May 25 11:46:19.290107 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.290014    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2brk\" (UniqueName: \"kubernetes.io/projected/d36a0111-eef5-423c-b113-22929de22f9a-kube-api-access-c2brk\") pod \"d36a0111-eef5-423c-b113-22929de22f9a\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") "
May 25 11:46:19.290107 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.290038    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d36a0111-eef5-423c-b113-22929de22f9a-tls-certs\") pod \"d36a0111-eef5-423c-b113-22929de22f9a\" (UID: \"d36a0111-eef5-423c-b113-22929de22f9a\") "
May 25 11:46:19.290369 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.290224    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d36a0111-eef5-423c-b113-22929de22f9a" (UID: "d36a0111-eef5-423c-b113-22929de22f9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:19.290593 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.290567    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-home" (OuterVolumeSpecName: "home") pod "d36a0111-eef5-423c-b113-22929de22f9a" (UID: "d36a0111-eef5-423c-b113-22929de22f9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:19.292495 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.292451    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d36a0111-eef5-423c-b113-22929de22f9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d36a0111-eef5-423c-b113-22929de22f9a" (UID: "d36a0111-eef5-423c-b113-22929de22f9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:46:19.292639 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.292623    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-dshm" (OuterVolumeSpecName: "dshm") pod "d36a0111-eef5-423c-b113-22929de22f9a" (UID: "d36a0111-eef5-423c-b113-22929de22f9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:19.292735 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.292716    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36a0111-eef5-423c-b113-22929de22f9a-kube-api-access-c2brk" (OuterVolumeSpecName: "kube-api-access-c2brk") pod "d36a0111-eef5-423c-b113-22929de22f9a" (UID: "d36a0111-eef5-423c-b113-22929de22f9a"). InnerVolumeSpecName "kube-api-access-c2brk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:46:19.312210 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.312169    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d36a0111-eef5-423c-b113-22929de22f9a" (UID: "d36a0111-eef5-423c-b113-22929de22f9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:19.358071 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.358032    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d36a0111-eef5-423c-b113-22929de22f9a" (UID: "d36a0111-eef5-423c-b113-22929de22f9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:46:19.391439 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.391399    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:46:19.391439 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.391433    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:46:19.391639 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.391446    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2brk\" (UniqueName: \"kubernetes.io/projected/d36a0111-eef5-423c-b113-22929de22f9a-kube-api-access-c2brk\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:46:19.391639 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.391477    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d36a0111-eef5-423c-b113-22929de22f9a-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:46:19.391639 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.391491    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:46:19.391639 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.391502    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:46:19.391639 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:19.391514    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d36a0111-eef5-423c-b113-22929de22f9a-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:46:20.120208 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:20.120173    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5579b55ddf-b726r_d36a0111-eef5-423c-b113-22929de22f9a/main/0.log"
May 25 11:46:20.120671 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:20.120604    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r" event={"ID":"d36a0111-eef5-423c-b113-22929de22f9a","Type":"ContainerDied","Data":"aece6ea616c3b77e94f2f2e9d7ff930d3b1072c4074f00700b69c156eb90248a"}
May 25 11:46:20.120671 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:20.120644    2578 scope.go:117] "RemoveContainer" containerID="cb67437902f70b98b334d84ae0a380caf9b9a8131f883056444a9a9bd18b6be9"
May 25 11:46:20.120787 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:20.120620    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"
May 25 11:46:20.129677 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:20.129659    2578 scope.go:117] "RemoveContainer" containerID="ffc60fd3eebdfad8904dbca9df030a730a5827dbd16bc6ca0799166d4d29e1d1"
May 25 11:46:20.145206 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:20.145175    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"]
May 25 11:46:20.149063 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:20.149035    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5579b55ddf-b726r"]
May 25 11:46:20.935603 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:20.935574    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d36a0111-eef5-423c-b113-22929de22f9a" path="/var/lib/kubelet/pods/d36a0111-eef5-423c-b113-22929de22f9a/volumes"
May 25 11:46:57.294809 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.294773    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"]
May 25 11:46:57.295293 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.295095    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="storage-initializer"
May 25 11:46:57.295293 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.295108    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="storage-initializer"
May 25 11:46:57.295293 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.295122    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main"
May 25 11:46:57.295293 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.295130    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main"
May 25 11:46:57.295293 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.295142    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="storage-initializer"
May 25 11:46:57.295293 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.295149    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="storage-initializer"
May 25 11:46:57.295293 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.295163    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main"
May 25 11:46:57.295293 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.295171    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main"
May 25 11:46:57.295293 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.295227    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="c6691d89-304c-495d-8cd5-cf31f0f4c679" containerName="main"
May 25 11:46:57.295293 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.295238    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="d36a0111-eef5-423c-b113-22929de22f9a" containerName="main"
May 25 11:46:57.297095 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.297075    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.300291 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.300268    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-5d6nj\""
May 25 11:46:57.300418 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.300328    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
May 25 11:46:57.312420 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.312389    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"]
May 25 11:46:57.413193 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.413158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.413193 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.413198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654c9dd8-9fec-4981-8a90-59ae16087eb2-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.413430 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.413225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.413430 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.413343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.413430 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.413414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.413584 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.413448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.413584 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.413507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.413584 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.413535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85kb9\" (UniqueName: \"kubernetes.io/projected/654c9dd8-9fec-4981-8a90-59ae16087eb2-kube-api-access-85kb9\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.413584 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.413566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.514899 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.514861    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515076 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.514912    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515076 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.515052    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515249 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.515101    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515249 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.515135    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-85kb9\" (UniqueName: \"kubernetes.io/projected/654c9dd8-9fec-4981-8a90-59ae16087eb2-kube-api-access-85kb9\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515249 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.515164    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515447 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.515259    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-workload-certs\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515447 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.515265    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515447 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.515305    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654c9dd8-9fec-4981-8a90-59ae16087eb2-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515447 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.515351    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515705 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.515493    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-workload-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515705 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.515671    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-data\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.515822 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.515805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-credential-socket\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.516056 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.516032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654c9dd8-9fec-4981-8a90-59ae16087eb2-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.517681 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.517662    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-envoy\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.517882 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.517864    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-podinfo\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.523108 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.523084    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/654c9dd8-9fec-4981-8a90-59ae16087eb2-istio-token\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.523406 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.523387    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85kb9\" (UniqueName: \"kubernetes.io/projected/654c9dd8-9fec-4981-8a90-59ae16087eb2-kube-api-access-85kb9\") pod \"router-gateway-2-openshift-default-7bb57db77b-bxmcp\" (UID: \"654c9dd8-9fec-4981-8a90-59ae16087eb2\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.610060 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.609958    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:46:57.755362 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:57.755329    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"]
May 25 11:46:57.756581 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:46:57.756557    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654c9dd8_9fec_4981_8a90_59ae16087eb2.slice/crio-e3c9c25fac9acae431e567b68f2067af24c9804a91c8a134c3ee6da74dc9c9ee WatchSource:0}: Error finding container e3c9c25fac9acae431e567b68f2067af24c9804a91c8a134c3ee6da74dc9c9ee: Status 404 returned error can't find the container with id e3c9c25fac9acae431e567b68f2067af24c9804a91c8a134c3ee6da74dc9c9ee
May 25 11:46:58.255871 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:46:58.255827    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp" event={"ID":"654c9dd8-9fec-4981-8a90-59ae16087eb2","Type":"ContainerStarted","Data":"e3c9c25fac9acae431e567b68f2067af24c9804a91c8a134c3ee6da74dc9c9ee"}
May 25 11:47:00.456055 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:00.456013    2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
May 25 11:47:00.456325 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:00.456107    2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
May 25 11:47:00.456325 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:00.456144    2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
May 25 11:47:01.269269 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:01.269230    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp" event={"ID":"654c9dd8-9fec-4981-8a90-59ae16087eb2","Type":"ContainerStarted","Data":"b8489faa1a9410224de8baa1911591de8c19c5c4b6723dba1ac16a3a7cb49192"}
May 25 11:47:01.292891 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:01.292834    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp" podStartSLOduration=1.595622208 podStartE2EDuration="4.29281855s" podCreationTimestamp="2026-05-25 11:46:57 +0000 UTC" firstStartedPulling="2026-05-25 11:46:57.758563562 +0000 UTC m=+1477.407848535" lastFinishedPulling="2026-05-25 11:47:00.45575989 +0000 UTC m=+1480.105044877" observedRunningTime="2026-05-25 11:47:01.290998724 +0000 UTC m=+1480.940283721" watchObservedRunningTime="2026-05-25 11:47:01.29281855 +0000 UTC m=+1480.942103544"
May 25 11:47:01.610325 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:01.610280    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:47:01.611562 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:01.611528    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp" podUID="654c9dd8-9fec-4981-8a90-59ae16087eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
May 25 11:47:02.610966 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:02.610914    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp" podUID="654c9dd8-9fec-4981-8a90-59ae16087eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
May 25 11:47:03.610683 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:03.610640    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp" podUID="654c9dd8-9fec-4981-8a90-59ae16087eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
May 25 11:47:04.615056 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:04.615023    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:47:04.615508 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:04.615349    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:47:04.617001 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:04.616982    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-7bb57db77b-bxmcp"
May 25 11:47:17.991733 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:17.991692    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"]
May 25 11:47:17.995135 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:17.995116    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:17.997930 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:17.997896    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-wdlrl\""
May 25 11:47:17.998086 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:17.997982    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 25 11:47:18.001678 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.001653    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"]
May 25 11:47:18.006247 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.006218    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.011878 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.011850    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"]
May 25 11:47:18.021507 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.021481    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"]
May 25 11:47:18.082361 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tmp-dir\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.082361 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-home\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.082628 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tls-certs\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.082628 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-home\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.082628 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.082628 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b84d557-bf21-4c6d-b3a7-5f0966522406-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.082628 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-dshm\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.082628 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5n7k\" (UniqueName: \"kubernetes.io/projected/1b84d557-bf21-4c6d-b3a7-5f0966522406-kube-api-access-d5n7k\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.082628 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-model-cache\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.082908 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.082908 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrpm\" (UniqueName: \"kubernetes.io/projected/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kube-api-access-2vrpm\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.082908 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.082908 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.082908 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.082751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.183810 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.183770    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.183810 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.183816    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tmp-dir\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.184036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.183836    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-home\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.184036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.183854    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tls-certs\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.184036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.183884    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-home\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.184036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.183911    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.184036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.183936    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b84d557-bf21-4c6d-b3a7-5f0966522406-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.184036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.183967    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-dshm\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.184036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184000    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d5n7k\" (UniqueName: \"kubernetes.io/projected/1b84d557-bf21-4c6d-b3a7-5f0966522406-kube-api-access-d5n7k\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.184036 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184027    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-model-cache\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.184444 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184060    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.184444 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184086    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrpm\" (UniqueName: \"kubernetes.io/projected/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kube-api-access-2vrpm\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.184444 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184118    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.184444 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184149    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.184444 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184243    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.184444 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184270    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tmp-dir\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.184444 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184328    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-home\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.184444 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184346    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.184444 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184417    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-home\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.184444 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184440    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.185007 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184528    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-model-cache\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.185007 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.184710    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.186617 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.186590    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-dshm\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.186740 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.186640    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.186740 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.186684    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tls-certs\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.187051 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.187034    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b84d557-bf21-4c6d-b3a7-5f0966522406-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.196939 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.196907    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5n7k\" (UniqueName: \"kubernetes.io/projected/1b84d557-bf21-4c6d-b3a7-5f0966522406-kube-api-access-d5n7k\") pod \"router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.197224 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.197207    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrpm\" (UniqueName: \"kubernetes.io/projected/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kube-api-access-2vrpm\") pod \"router-with-refs-pd-test-kserve-55c49d68cf-mtk62\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.305803 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.305707    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:18.318644 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.318579    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:47:18.451344 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.451318    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"]
May 25 11:47:18.454285 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:47:18.454257    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c80fd7_9a8e_4460_b7a1_c69cd63aa3bc.slice/crio-b09cde10b1a5b969f4b0bb1a112d81e11bb4299e24b334043a0ec2a7e6fd922e WatchSource:0}: Error finding container b09cde10b1a5b969f4b0bb1a112d81e11bb4299e24b334043a0ec2a7e6fd922e: Status 404 returned error can't find the container with id b09cde10b1a5b969f4b0bb1a112d81e11bb4299e24b334043a0ec2a7e6fd922e
May 25 11:47:18.475863 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:18.475835    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"]
May 25 11:47:18.476292 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:47:18.476265    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b84d557_bf21_4c6d_b3a7_5f0966522406.slice/crio-4bc1c0f0b3c0ecc8f1677bb6651858df1a59693f67924eeabcdffdbd5f06dd66 WatchSource:0}: Error finding container 4bc1c0f0b3c0ecc8f1677bb6651858df1a59693f67924eeabcdffdbd5f06dd66: Status 404 returned error can't find the container with id 4bc1c0f0b3c0ecc8f1677bb6651858df1a59693f67924eeabcdffdbd5f06dd66
May 25 11:47:19.333572 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:19.333540    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" event={"ID":"1b84d557-bf21-4c6d-b3a7-5f0966522406","Type":"ContainerStarted","Data":"71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702"}
May 25 11:47:19.334006 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:19.333583    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" event={"ID":"1b84d557-bf21-4c6d-b3a7-5f0966522406","Type":"ContainerStarted","Data":"4bc1c0f0b3c0ecc8f1677bb6651858df1a59693f67924eeabcdffdbd5f06dd66"}
May 25 11:47:19.335329 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:19.335304    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" event={"ID":"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc","Type":"ContainerStarted","Data":"4a7a414e95e742f35037645dca33ba327dbc54f283ccd81ab580489645ac176a"}
May 25 11:47:19.335451 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:19.335337    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" event={"ID":"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc","Type":"ContainerStarted","Data":"b09cde10b1a5b969f4b0bb1a112d81e11bb4299e24b334043a0ec2a7e6fd922e"}
May 25 11:47:19.335451 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:19.335414    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:47:20.343254 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:20.343197    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" event={"ID":"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc","Type":"ContainerStarted","Data":"1b03c2e4ce37a84a522141e5e0a5cdbe8a7afb3d3aa4a51e160283ff1be504ce"}
May 25 11:47:20.996146 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:20.996115    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:47:20.997103 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:20.997075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:47:31.361181 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:47:31.361148    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 11:48:30.606800 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:48:30.606767    2578 generic.go:358] "Generic (PLEG): container finished" podID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerID="71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702" exitCode=0
May 25 11:48:30.607226 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:48:30.606848    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" event={"ID":"1b84d557-bf21-4c6d-b3a7-5f0966522406","Type":"ContainerDied","Data":"71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702"}
May 25 11:48:31.613206 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:48:31.613166    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" event={"ID":"1b84d557-bf21-4c6d-b3a7-5f0966522406","Type":"ContainerStarted","Data":"ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d"}
May 25 11:48:31.642563 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:48:31.642504    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" podStartSLOduration=74.642484677 podStartE2EDuration="1m14.642484677s" podCreationTimestamp="2026-05-25 11:47:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:48:31.63913199 +0000 UTC m=+1571.288416988" watchObservedRunningTime="2026-05-25 11:48:31.642484677 +0000 UTC m=+1571.291769674"
May 25 11:48:38.318809 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:48:38.318765    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:48:38.318809 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:48:38.318810    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:48:38.320525 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:48:38.320497    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
May 25 11:48:48.319692 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:48:48.319641    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
May 25 11:48:58.319364 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:48:58.319315    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
May 25 11:49:08.319642 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:49:08.319543    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
May 25 11:49:18.318989 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:49:18.318943    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
May 25 11:49:28.319407 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:49:28.319365    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
May 25 11:49:38.319772 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:49:38.319722    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
May 25 11:49:48.319876 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:49:48.319832    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
May 25 11:49:58.319892 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:49:58.319847    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
May 25 11:50:08.328627 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:50:08.328593    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:50:08.336447 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:50:08.336427    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 11:52:21.026677 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:52:21.026649    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:52:21.029203 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:52:21.029178    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:57:21.055953 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:57:21.055924    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:57:21.060346 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:57:21.060325    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 11:59:20.030374 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:20.030335    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"]
May 25 11:59:20.031008 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:20.030628    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk" podUID="677e8234-983d-43c6-a498-56ae63d43f2c" containerName="storage-initializer" containerID="cri-o://b414534c4400ea0dbb7041e5ee78c3bcd8f065eb2bcff94d47806aa3101e8c2a" gracePeriod=30
May 25 11:59:27.532562 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.532531    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"]
May 25 11:59:27.538089 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.538059    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"]
May 25 11:59:27.538254 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.538202    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.541139 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.541113    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 25 11:59:27.541276 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.541219    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-snm9p\""
May 25 11:59:27.541981 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.541963    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.552116 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.552090    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"]
May 25 11:59:27.556309 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.556202    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"]
May 25 11:59:27.693535 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.693709 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.693709 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.693709 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.693709 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.693709 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.693885 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff253078-cbf6-4087-b706-0abfd2da87f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.693885 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9swm\" (UniqueName: \"kubernetes.io/projected/ff253078-cbf6-4087-b706-0abfd2da87f6-kube-api-access-b9swm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.693885 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2m7\" (UniqueName: \"kubernetes.io/projected/f059415b-e1a6-4811-8d31-470b7703d996-kube-api-access-sh2m7\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.693885 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.693885 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.693885 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.694081 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.694081 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.693920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f059415b-e1a6-4811-8d31-470b7703d996-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.794602 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.794516    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.794602 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.794575    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff253078-cbf6-4087-b706-0abfd2da87f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.794832 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.794606    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b9swm\" (UniqueName: \"kubernetes.io/projected/ff253078-cbf6-4087-b706-0abfd2da87f6-kube-api-access-b9swm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.794832 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.794643    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2m7\" (UniqueName: \"kubernetes.io/projected/f059415b-e1a6-4811-8d31-470b7703d996-kube-api-access-sh2m7\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.794832 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.794671    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.794832 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.794700    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.794832 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.794720    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.795149 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.794906    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.795149 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795085    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.795149 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795097    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f059415b-e1a6-4811-8d31-470b7703d996-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.795283 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795185    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.795283 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795234    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.795363 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795333    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.795418 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795378    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.795418 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795406    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.795570 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795488    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.795570 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795532    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.795681 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795640    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.795762 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795738    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.795910 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.795890    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.796034 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.796013    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.796075 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.796051    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.797186 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.797166    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.797661 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.797637    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f059415b-e1a6-4811-8d31-470b7703d996-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.797754 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.797672    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.797830 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.797811    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff253078-cbf6-4087-b706-0abfd2da87f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.807858 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.807828    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9swm\" (UniqueName: \"kubernetes.io/projected/ff253078-cbf6-4087-b706-0abfd2da87f6-kube-api-access-b9swm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.808252 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.808233    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2m7\" (UniqueName: \"kubernetes.io/projected/f059415b-e1a6-4811-8d31-470b7703d996-kube-api-access-sh2m7\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:27.851841 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.851803    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:27.860399 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:27.860169    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:28.001903 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:28.001873    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"]
May 25 11:59:28.002908 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:59:28.002879    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff253078_cbf6_4087_b706_0abfd2da87f6.slice/crio-7a88dfb34ce4e973c84fba0a4b7afae9352e5b35b875f1ad75668b2016bc1c2f WatchSource:0}: Error finding container 7a88dfb34ce4e973c84fba0a4b7afae9352e5b35b875f1ad75668b2016bc1c2f: Status 404 returned error can't find the container with id 7a88dfb34ce4e973c84fba0a4b7afae9352e5b35b875f1ad75668b2016bc1c2f
May 25 11:59:28.004891 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:28.004871    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:59:28.028580 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:28.028555    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"]
May 25 11:59:28.033613 ip-10-0-133-58 kubenswrapper[2578]: W0525 11:59:28.033589    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf059415b_e1a6_4811_8d31_470b7703d996.slice/crio-5e7ce4fd38129f7b1dd307338177a85b7fd475969eaf6ff0de9e502f2751eb1c WatchSource:0}: Error finding container 5e7ce4fd38129f7b1dd307338177a85b7fd475969eaf6ff0de9e502f2751eb1c: Status 404 returned error can't find the container with id 5e7ce4fd38129f7b1dd307338177a85b7fd475969eaf6ff0de9e502f2751eb1c
May 25 11:59:28.107667 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:28.107612    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" event={"ID":"ff253078-cbf6-4087-b706-0abfd2da87f6","Type":"ContainerStarted","Data":"7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5"}
May 25 11:59:28.107667 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:28.107665    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" event={"ID":"ff253078-cbf6-4087-b706-0abfd2da87f6","Type":"ContainerStarted","Data":"7a88dfb34ce4e973c84fba0a4b7afae9352e5b35b875f1ad75668b2016bc1c2f"}
May 25 11:59:28.107921 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:28.107808    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:28.109320 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:28.109292    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" event={"ID":"f059415b-e1a6-4811-8d31-470b7703d996","Type":"ContainerStarted","Data":"3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f"}
May 25 11:59:28.109410 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:28.109328    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" event={"ID":"f059415b-e1a6-4811-8d31-470b7703d996","Type":"ContainerStarted","Data":"5e7ce4fd38129f7b1dd307338177a85b7fd475969eaf6ff0de9e502f2751eb1c"}
May 25 11:59:29.116154 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:29.116096    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" event={"ID":"ff253078-cbf6-4087-b706-0abfd2da87f6","Type":"ContainerStarted","Data":"865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792"}
May 25 11:59:35.142729 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:35.142695    2578 generic.go:358] "Generic (PLEG): container finished" podID="f059415b-e1a6-4811-8d31-470b7703d996" containerID="3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f" exitCode=0
May 25 11:59:35.143167 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:35.142779    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" event={"ID":"f059415b-e1a6-4811-8d31-470b7703d996","Type":"ContainerDied","Data":"3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f"}
May 25 11:59:36.147576 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:36.147539    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" event={"ID":"f059415b-e1a6-4811-8d31-470b7703d996","Type":"ContainerStarted","Data":"3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d"}
May 25 11:59:36.175103 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:36.175049    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podStartSLOduration=9.175033472 podStartE2EDuration="9.175033472s" podCreationTimestamp="2026-05-25 11:59:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:59:36.171761461 +0000 UTC m=+2235.821046455" watchObservedRunningTime="2026-05-25 11:59:36.175033472 +0000 UTC m=+2235.824318467"
May 25 11:59:37.861198 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:37.861164    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:37.861602 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:37.861209    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 11:59:37.862933 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:37.862902    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 25 11:59:40.135172 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:40.135135    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 11:59:47.860932 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:47.860885    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 25 11:59:50.205179 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.205153    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-7f9485d6b5-jw4rk_677e8234-983d-43c6-a498-56ae63d43f2c/storage-initializer/0.log"
May 25 11:59:50.205574 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.205194    2578 generic.go:358] "Generic (PLEG): container finished" podID="677e8234-983d-43c6-a498-56ae63d43f2c" containerID="b414534c4400ea0dbb7041e5ee78c3bcd8f065eb2bcff94d47806aa3101e8c2a" exitCode=137
May 25 11:59:50.205574 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.205288    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk" event={"ID":"677e8234-983d-43c6-a498-56ae63d43f2c","Type":"ContainerDied","Data":"b414534c4400ea0dbb7041e5ee78c3bcd8f065eb2bcff94d47806aa3101e8c2a"}
May 25 11:59:50.249712 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.249689    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-7f9485d6b5-jw4rk_677e8234-983d-43c6-a498-56ae63d43f2c/storage-initializer/0.log"
May 25 11:59:50.249850 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.249752    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:59:50.301400 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.301312    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/677e8234-983d-43c6-a498-56ae63d43f2c-tls-certs\") pod \"677e8234-983d-43c6-a498-56ae63d43f2c\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") "
May 25 11:59:50.301400 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.301367    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dhsn\" (UniqueName: \"kubernetes.io/projected/677e8234-983d-43c6-a498-56ae63d43f2c-kube-api-access-2dhsn\") pod \"677e8234-983d-43c6-a498-56ae63d43f2c\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") "
May 25 11:59:50.301400 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.301386    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-kserve-provision-location\") pod \"677e8234-983d-43c6-a498-56ae63d43f2c\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") "
May 25 11:59:50.301400 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.301402    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-home\") pod \"677e8234-983d-43c6-a498-56ae63d43f2c\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") "
May 25 11:59:50.301776 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.301488    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-tmp-dir\") pod \"677e8234-983d-43c6-a498-56ae63d43f2c\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") "
May 25 11:59:50.301776 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.301518    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-dshm\") pod \"677e8234-983d-43c6-a498-56ae63d43f2c\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") "
May 25 11:59:50.301776 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.301543    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-model-cache\") pod \"677e8234-983d-43c6-a498-56ae63d43f2c\" (UID: \"677e8234-983d-43c6-a498-56ae63d43f2c\") "
May 25 11:59:50.301930 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.301778    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-home" (OuterVolumeSpecName: "home") pod "677e8234-983d-43c6-a498-56ae63d43f2c" (UID: "677e8234-983d-43c6-a498-56ae63d43f2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:59:50.301930 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.301793    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "677e8234-983d-43c6-a498-56ae63d43f2c" (UID: "677e8234-983d-43c6-a498-56ae63d43f2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:59:50.302038 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.301973    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "677e8234-983d-43c6-a498-56ae63d43f2c" (UID: "677e8234-983d-43c6-a498-56ae63d43f2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:59:50.303737 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.303696    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677e8234-983d-43c6-a498-56ae63d43f2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "677e8234-983d-43c6-a498-56ae63d43f2c" (UID: "677e8234-983d-43c6-a498-56ae63d43f2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:59:50.303916 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.303896    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677e8234-983d-43c6-a498-56ae63d43f2c-kube-api-access-2dhsn" (OuterVolumeSpecName: "kube-api-access-2dhsn") pod "677e8234-983d-43c6-a498-56ae63d43f2c" (UID: "677e8234-983d-43c6-a498-56ae63d43f2c"). InnerVolumeSpecName "kube-api-access-2dhsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:59:50.304260 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.304229    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-dshm" (OuterVolumeSpecName: "dshm") pod "677e8234-983d-43c6-a498-56ae63d43f2c" (UID: "677e8234-983d-43c6-a498-56ae63d43f2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:59:50.317290 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.317256    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "677e8234-983d-43c6-a498-56ae63d43f2c" (UID: "677e8234-983d-43c6-a498-56ae63d43f2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:59:50.403153 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.403105    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:59:50.403153 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.403155    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:59:50.403369 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.403171    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:59:50.403369 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.403187    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/677e8234-983d-43c6-a498-56ae63d43f2c-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:59:50.403369 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.403203    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dhsn\" (UniqueName: \"kubernetes.io/projected/677e8234-983d-43c6-a498-56ae63d43f2c-kube-api-access-2dhsn\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:59:50.403369 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.403219    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:59:50.403369 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:50.403233    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/677e8234-983d-43c6-a498-56ae63d43f2c-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 11:59:51.210568 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:51.210481    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-7f9485d6b5-jw4rk_677e8234-983d-43c6-a498-56ae63d43f2c/storage-initializer/0.log"
May 25 11:59:51.211009 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:51.210562    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk" event={"ID":"677e8234-983d-43c6-a498-56ae63d43f2c","Type":"ContainerDied","Data":"bb7f41c9555f0b3e10a18e02efda3bb99ed9e101b3a992baa8e1712e3bad5b53"}
May 25 11:59:51.211009 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:51.210602    2578 scope.go:117] "RemoveContainer" containerID="b414534c4400ea0dbb7041e5ee78c3bcd8f065eb2bcff94d47806aa3101e8c2a"
May 25 11:59:51.211009 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:51.210603    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"
May 25 11:59:51.247029 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:51.246992    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"]
May 25 11:59:51.254276 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:51.254240    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-7f9485d6b5-jw4rk"]
May 25 11:59:52.935777 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:52.935743    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="677e8234-983d-43c6-a498-56ae63d43f2c" path="/var/lib/kubelet/pods/677e8234-983d-43c6-a498-56ae63d43f2c/volumes"
May 25 11:59:57.861391 ip-10-0-133-58 kubenswrapper[2578]: I0525 11:59:57.861330    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 25 12:00:07.861155 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:07.861101    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 25 12:00:17.861762 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:17.861721    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 25 12:00:23.337809 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:23.337714    2578 generic.go:358] "Generic (PLEG): container finished" podID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerID="865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792" exitCode=0
May 25 12:00:23.337809 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:23.337787    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" event={"ID":"ff253078-cbf6-4087-b706-0abfd2da87f6","Type":"ContainerDied","Data":"865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792"}
May 25 12:00:24.343932 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:24.343897    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" event={"ID":"ff253078-cbf6-4087-b706-0abfd2da87f6","Type":"ContainerStarted","Data":"706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e"}
May 25 12:00:24.372542 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:24.372484    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podStartSLOduration=57.372439885 podStartE2EDuration="57.372439885s" podCreationTimestamp="2026-05-25 11:59:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:00:24.369174279 +0000 UTC m=+2284.018459287" watchObservedRunningTime="2026-05-25 12:00:24.372439885 +0000 UTC m=+2284.021724882"
May 25 12:00:27.852614 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:27.852572    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 12:00:27.853084 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:27.852770    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 12:00:27.853176 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:27.853140    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
May 25 12:00:27.861186 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:27.861134    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 25 12:00:37.853198 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:37.853147    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
May 25 12:00:37.861225 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:37.861182    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 25 12:00:47.852784 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:47.852712    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
May 25 12:00:47.860825 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:47.860782    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 25 12:00:57.853025 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:57.852971    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
May 25 12:00:57.861674 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:00:57.861634    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 25 12:01:07.853262 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:01:07.853205    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
May 25 12:01:07.861743 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:01:07.861710    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 25 12:01:17.852698 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:01:17.852639    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
May 25 12:01:17.874119 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:01:17.874090    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 12:01:17.881933 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:01:17.881904    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 12:01:27.852830 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:01:27.852767    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
May 25 12:01:37.852995 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:01:37.852944    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
May 25 12:01:47.852675 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:01:47.852624    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
May 25 12:01:57.853204 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:01:57.853161    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
May 25 12:02:01.342003 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:01.341964    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"]
May 25 12:02:01.342565 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:01.342231    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main" containerID="cri-o://ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d" gracePeriod=30
May 25 12:02:01.348212 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:01.348183    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"]
May 25 12:02:01.348542 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:01.348517    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="storage-initializer" containerID="cri-o://1b03c2e4ce37a84a522141e5e0a5cdbe8a7afb3d3aa4a51e160283ff1be504ce" gracePeriod=30
May 25 12:02:01.348643 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:01.348517    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="llm-d-routing-sidecar" containerID="cri-o://4a7a414e95e742f35037645dca33ba327dbc54f283ccd81ab580489645ac176a" gracePeriod=30
May 25 12:02:01.353676 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:01.353635    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": EOF"
May 25 12:02:01.756096 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:01.756063    2578 generic.go:358] "Generic (PLEG): container finished" podID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerID="4a7a414e95e742f35037645dca33ba327dbc54f283ccd81ab580489645ac176a" exitCode=0
May 25 12:02:01.756280 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:01.756145    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" event={"ID":"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc","Type":"ContainerDied","Data":"4a7a414e95e742f35037645dca33ba327dbc54f283ccd81ab580489645ac176a"}
May 25 12:02:07.423229 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.423195    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"]
May 25 12:02:07.423694 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.423566    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="677e8234-983d-43c6-a498-56ae63d43f2c" containerName="storage-initializer"
May 25 12:02:07.423694 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.423581    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="677e8234-983d-43c6-a498-56ae63d43f2c" containerName="storage-initializer"
May 25 12:02:07.423694 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.423676    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="677e8234-983d-43c6-a498-56ae63d43f2c" containerName="storage-initializer"
May 25 12:02:07.426927 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.426907    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.429934 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.429910    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
May 25 12:02:07.439884 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.439859    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"]
May 25 12:02:07.464573 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.464543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.464573 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.464583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.464802 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.464612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.464802 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.464632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ada20ec-3961-4c62-9c41-55d7a22a519d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.464802 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.464664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.464802 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.464695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7glx\" (UniqueName: \"kubernetes.io/projected/0ada20ec-3961-4c62-9c41-55d7a22a519d-kube-api-access-l7glx\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.464802 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.464739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.566106 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.566058    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.566106 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.566103    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.566368 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.566136    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.566368 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.566158    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ada20ec-3961-4c62-9c41-55d7a22a519d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.566368 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.566186    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.566368 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.566233    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l7glx\" (UniqueName: \"kubernetes.io/projected/0ada20ec-3961-4c62-9c41-55d7a22a519d-kube-api-access-l7glx\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.566368 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.566264    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.566712 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.566677    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.566797 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.566715    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.566797 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.566751    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.567180 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.567151    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.568736 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.568715    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.568913 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.568892    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ada20ec-3961-4c62-9c41-55d7a22a519d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.575159 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.575132    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7glx\" (UniqueName: \"kubernetes.io/projected/0ada20ec-3961-4c62-9c41-55d7a22a519d-kube-api-access-l7glx\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.737209 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.737162    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:02:07.864841 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.864809    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 12:02:07.876518 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.876493    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 12:02:07.884378 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:07.884349    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"]
May 25 12:02:08.307075 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:08.307033    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
May 25 12:02:08.785542 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:08.785500    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" event={"ID":"0ada20ec-3961-4c62-9c41-55d7a22a519d","Type":"ContainerStarted","Data":"80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6"}
May 25 12:02:08.785542 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:08.785543    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" event={"ID":"0ada20ec-3961-4c62-9c41-55d7a22a519d","Type":"ContainerStarted","Data":"fe2b84664f186624db1870f6d4d1e1c80e14d83f1d35a0818df6e32539564034"}
May 25 12:02:11.347702 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:11.347662    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
May 25 12:02:18.307269 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:18.307204    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
May 25 12:02:21.091864 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:21.091830    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 12:02:21.102765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:21.102728    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 12:02:21.348107 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:21.348019    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
May 25 12:02:28.306908 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:28.306855    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
May 25 12:02:28.307312 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:28.306941    2578 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 12:02:31.009034 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.008994    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"]
May 25 12:02:31.009440 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.009327    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main" containerID="cri-o://3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d" gracePeriod=30
May 25 12:02:31.027056 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.027017    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"]
May 25 12:02:31.027532 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.027466    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main" containerID="cri-o://706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e" gracePeriod=30
May 25 12:02:31.348437 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.348403    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
May 25 12:02:31.597885 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.597821    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 12:02:31.691734 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.691704    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-home\") pod \"1b84d557-bf21-4c6d-b3a7-5f0966522406\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") "
May 25 12:02:31.691929 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.691760    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-model-cache\") pod \"1b84d557-bf21-4c6d-b3a7-5f0966522406\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") "
May 25 12:02:31.691929 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.691786    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5n7k\" (UniqueName: \"kubernetes.io/projected/1b84d557-bf21-4c6d-b3a7-5f0966522406-kube-api-access-d5n7k\") pod \"1b84d557-bf21-4c6d-b3a7-5f0966522406\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") "
May 25 12:02:31.691929 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.691816    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-tmp-dir\") pod \"1b84d557-bf21-4c6d-b3a7-5f0966522406\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") "
May 25 12:02:31.691929 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.691847    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b84d557-bf21-4c6d-b3a7-5f0966522406-tls-certs\") pod \"1b84d557-bf21-4c6d-b3a7-5f0966522406\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") "
May 25 12:02:31.691929 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.691863    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-kserve-provision-location\") pod \"1b84d557-bf21-4c6d-b3a7-5f0966522406\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") "
May 25 12:02:31.691929 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.691894    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-dshm\") pod \"1b84d557-bf21-4c6d-b3a7-5f0966522406\" (UID: \"1b84d557-bf21-4c6d-b3a7-5f0966522406\") "
May 25 12:02:31.692252 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.692095    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b84d557-bf21-4c6d-b3a7-5f0966522406" (UID: "1b84d557-bf21-4c6d-b3a7-5f0966522406"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:02:31.692311 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.692262    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:31.692442 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.692418    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-home" (OuterVolumeSpecName: "home") pod "1b84d557-bf21-4c6d-b3a7-5f0966522406" (UID: "1b84d557-bf21-4c6d-b3a7-5f0966522406"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:02:31.694321 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.694296    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-dshm" (OuterVolumeSpecName: "dshm") pod "1b84d557-bf21-4c6d-b3a7-5f0966522406" (UID: "1b84d557-bf21-4c6d-b3a7-5f0966522406"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:02:31.694450 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.694373    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b84d557-bf21-4c6d-b3a7-5f0966522406-kube-api-access-d5n7k" (OuterVolumeSpecName: "kube-api-access-d5n7k") pod "1b84d557-bf21-4c6d-b3a7-5f0966522406" (UID: "1b84d557-bf21-4c6d-b3a7-5f0966522406"). InnerVolumeSpecName "kube-api-access-d5n7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:02:31.695030 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.695005    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b84d557-bf21-4c6d-b3a7-5f0966522406-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b84d557-bf21-4c6d-b3a7-5f0966522406" (UID: "1b84d557-bf21-4c6d-b3a7-5f0966522406"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:02:31.707706 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.707666    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b84d557-bf21-4c6d-b3a7-5f0966522406" (UID: "1b84d557-bf21-4c6d-b3a7-5f0966522406"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:02:31.753131 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.753070    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b84d557-bf21-4c6d-b3a7-5f0966522406" (UID: "1b84d557-bf21-4c6d-b3a7-5f0966522406"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:02:31.793135 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.793107    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5n7k\" (UniqueName: \"kubernetes.io/projected/1b84d557-bf21-4c6d-b3a7-5f0966522406-kube-api-access-d5n7k\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:31.793135 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.793134    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:31.793339 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.793146    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b84d557-bf21-4c6d-b3a7-5f0966522406-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:31.793339 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.793155    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:31.793339 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.793163    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:31.793339 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.793171    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b84d557-bf21-4c6d-b3a7-5f0966522406-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:31.889506 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.889442    2578 generic.go:358] "Generic (PLEG): container finished" podID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerID="ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d" exitCode=137
May 25 12:02:31.889685 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.889520    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" event={"ID":"1b84d557-bf21-4c6d-b3a7-5f0966522406","Type":"ContainerDied","Data":"ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d"}
May 25 12:02:31.889685 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.889560    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf" event={"ID":"1b84d557-bf21-4c6d-b3a7-5f0966522406","Type":"ContainerDied","Data":"4bc1c0f0b3c0ecc8f1677bb6651858df1a59693f67924eeabcdffdbd5f06dd66"}
May 25 12:02:31.889685 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.889572    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"
May 25 12:02:31.889685 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.889581    2578 scope.go:117] "RemoveContainer" containerID="ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d"
May 25 12:02:31.891225 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.891193    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-55c49d68cf-mtk62_13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc/storage-initializer/0.log"
May 25 12:02:31.891582 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.891562    2578 generic.go:358] "Generic (PLEG): container finished" podID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerID="1b03c2e4ce37a84a522141e5e0a5cdbe8a7afb3d3aa4a51e160283ff1be504ce" exitCode=137
May 25 12:02:31.891681 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.891618    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" event={"ID":"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc","Type":"ContainerDied","Data":"1b03c2e4ce37a84a522141e5e0a5cdbe8a7afb3d3aa4a51e160283ff1be504ce"}
May 25 12:02:31.900485 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.900440    2578 scope.go:117] "RemoveContainer" containerID="71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702"
May 25 12:02:31.911843 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.911812    2578 scope.go:117] "RemoveContainer" containerID="ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d"
May 25 12:02:31.912732 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:02:31.912700    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d\": container with ID starting with ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d not found: ID does not exist" containerID="ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d"
May 25 12:02:31.912813 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.912750    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d"} err="failed to get container status \"ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d\": rpc error: code = NotFound desc = could not find container \"ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d\": container with ID starting with ef3c303f6aeba74b7cd478cabff5daabf8166fe6719ff58c9490ec7e5ee9311d not found: ID does not exist"
May 25 12:02:31.912813 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.912779    2578 scope.go:117] "RemoveContainer" containerID="71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702"
May 25 12:02:31.913161 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:02:31.913141    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702\": container with ID starting with 71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702 not found: ID does not exist" containerID="71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702"
May 25 12:02:31.913242 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.913181    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702"} err="failed to get container status \"71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702\": rpc error: code = NotFound desc = could not find container \"71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702\": container with ID starting with 71151e45fe2b781aceea233014fd3020678a4815e16b1e8593a6488f13bbf702 not found: ID does not exist"
May 25 12:02:31.915481 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.915438    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"]
May 25 12:02:31.917897 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:31.917872    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-6dc8566f8-xx9gf"]
May 25 12:02:32.010077 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.010055    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-55c49d68cf-mtk62_13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc/storage-initializer/0.log"
May 25 12:02:32.010521 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.010500    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 12:02:32.095779 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.095742    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-home\") pod \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") "
May 25 12:02:32.095988 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.095790    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tls-certs\") pod \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") "
May 25 12:02:32.095988 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.095838    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kserve-provision-location\") pod \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") "
May 25 12:02:32.095988 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.095883    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tmp-dir\") pod \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") "
May 25 12:02:32.095988 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.095902    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-dshm\") pod \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") "
May 25 12:02:32.096209 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.096044    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vrpm\" (UniqueName: \"kubernetes.io/projected/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kube-api-access-2vrpm\") pod \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") "
May 25 12:02:32.096209 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.096056    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-home" (OuterVolumeSpecName: "home") pod "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" (UID: "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:02:32.096209 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.096083    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-model-cache\") pod \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\" (UID: \"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc\") "
May 25 12:02:32.096209 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.096109    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" (UID: "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:02:32.096365 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.096333    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" (UID: "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:02:32.096365 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.096351    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:32.096491 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.096371    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:32.098251 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.098229    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-dshm" (OuterVolumeSpecName: "dshm") pod "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" (UID: "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:02:32.098384 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.098299    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" (UID: "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:02:32.098384 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.098338    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kube-api-access-2vrpm" (OuterVolumeSpecName: "kube-api-access-2vrpm") pod "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" (UID: "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc"). InnerVolumeSpecName "kube-api-access-2vrpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:02:32.133813 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.133728    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" (UID: "13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:02:32.196968 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.196940    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:32.196968 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.196967    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vrpm\" (UniqueName: \"kubernetes.io/projected/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kube-api-access-2vrpm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:32.197160 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.196979    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:32.197160 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.196988    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:32.197160 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.196997    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:02:32.897621 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.897590    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-55c49d68cf-mtk62_13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc/storage-initializer/0.log"
May 25 12:02:32.897961 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.897939    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62" event={"ID":"13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc","Type":"ContainerDied","Data":"b09cde10b1a5b969f4b0bb1a112d81e11bb4299e24b334043a0ec2a7e6fd922e"}
May 25 12:02:32.898014 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.897983    2578 scope.go:117] "RemoveContainer" containerID="1b03c2e4ce37a84a522141e5e0a5cdbe8a7afb3d3aa4a51e160283ff1be504ce"
May 25 12:02:32.898014 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.897995    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"
May 25 12:02:32.922208 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.922188    2578 scope.go:117] "RemoveContainer" containerID="4a7a414e95e742f35037645dca33ba327dbc54f283ccd81ab580489645ac176a"
May 25 12:02:32.936282 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.936252    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" path="/var/lib/kubelet/pods/1b84d557-bf21-4c6d-b3a7-5f0966522406/volumes"
May 25 12:02:32.940985 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.940874    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"]
May 25 12:02:32.943195 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:32.943169    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-55c49d68cf-mtk62"]
May 25 12:02:34.934840 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:34.934805    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" path="/var/lib/kubelet/pods/13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc/volumes"
May 25 12:02:40.646181 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646088    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"]
May 25 12:02:40.646682 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646510    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="storage-initializer"
May 25 12:02:40.646682 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646525    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="storage-initializer"
May 25 12:02:40.646682 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646539    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="storage-initializer"
May 25 12:02:40.646682 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646545    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="storage-initializer"
May 25 12:02:40.646682 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646552    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="llm-d-routing-sidecar"
May 25 12:02:40.646682 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646559    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="llm-d-routing-sidecar"
May 25 12:02:40.646682 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646573    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main"
May 25 12:02:40.646682 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646612    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main"
May 25 12:02:40.647336 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646742    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="storage-initializer"
May 25 12:02:40.647336 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646755    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="1b84d557-bf21-4c6d-b3a7-5f0966522406" containerName="main"
May 25 12:02:40.647336 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.646762    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="13c80fd7-9a8e-4460-b7a1-c69cd63aa3bc" containerName="llm-d-routing-sidecar"
May 25 12:02:40.652768 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.652742    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.657065 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.657027    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-dcvpf\""
May 25 12:02:40.657212 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.657086    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 25 12:02:40.658347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.658312    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"]
May 25 12:02:40.661858 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.661837    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.668154 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.668291 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.668291 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-dshm\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.668291 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-model-cache\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.668422 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.668422 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.668422 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.668570 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36388f0e-926b-4e51-8018-93324ff2b456-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.668570 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t55vf\" (UniqueName: \"kubernetes.io/projected/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kube-api-access-t55vf\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.668570 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668527    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqfvv\" (UniqueName: \"kubernetes.io/projected/36388f0e-926b-4e51-8018-93324ff2b456-kube-api-access-cqfvv\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.668689 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.668689 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.668689 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-home\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.668689 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.668656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.669803 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.669782    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"]
May 25 12:02:40.677500 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.677477    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"]
May 25 12:02:40.769530 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769495    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-model-cache\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.769530 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769536    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.769756 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769555    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.769756 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769681    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.769756 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769738    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36388f0e-926b-4e51-8018-93324ff2b456-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.769924 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769769    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t55vf\" (UniqueName: \"kubernetes.io/projected/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kube-api-access-t55vf\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.769924 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769804    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cqfvv\" (UniqueName: \"kubernetes.io/projected/36388f0e-926b-4e51-8018-93324ff2b456-kube-api-access-cqfvv\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.769924 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769836    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.769924 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769869    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.769924 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769886    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.769924 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769899    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-home\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.770241 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769927    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.770241 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769944    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-model-cache\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.770241 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.769971    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.770241 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.770036    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.770241 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.770089    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.770241 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.770124    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.770241 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.770178    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-dshm\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.770241 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.770211    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-home\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.770656 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.770382    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.770656 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.770426    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.770656 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.770534    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.772574 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.772545    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-dshm\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.772711 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.772553    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.772950 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.772931    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36388f0e-926b-4e51-8018-93324ff2b456-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.773030 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.772992    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.779592 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.779567    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqfvv\" (UniqueName: \"kubernetes.io/projected/36388f0e-926b-4e51-8018-93324ff2b456-kube-api-access-cqfvv\") pod \"custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:40.779702 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.779629    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t55vf\" (UniqueName: \"kubernetes.io/projected/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kube-api-access-t55vf\") pod \"custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.964044 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.963954    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:40.974354 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:40.974317    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:41.115347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:41.115321    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"]
May 25 12:02:41.117345 ip-10-0-133-58 kubenswrapper[2578]: W0525 12:02:41.117315    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8fbc35_5e88_4229_af1e_64c5dedd62f3.slice/crio-ec2f2aa609ca8de46f534d10e361c883ab36446a133eee5366c0a52a5c97ca19 WatchSource:0}: Error finding container ec2f2aa609ca8de46f534d10e361c883ab36446a133eee5366c0a52a5c97ca19: Status 404 returned error can't find the container with id ec2f2aa609ca8de46f534d10e361c883ab36446a133eee5366c0a52a5c97ca19
May 25 12:02:41.138194 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:41.138166    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"]
May 25 12:02:41.139278 ip-10-0-133-58 kubenswrapper[2578]: W0525 12:02:41.139252    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36388f0e_926b_4e51_8018_93324ff2b456.slice/crio-0468ac7b459f8f747094a9b3a726a69391f6ab2e44c34a6961b749391ddd219d WatchSource:0}: Error finding container 0468ac7b459f8f747094a9b3a726a69391f6ab2e44c34a6961b749391ddd219d: Status 404 returned error can't find the container with id 0468ac7b459f8f747094a9b3a726a69391f6ab2e44c34a6961b749391ddd219d
May 25 12:02:41.934225 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:41.934183    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" event={"ID":"3f8fbc35-5e88-4229-af1e-64c5dedd62f3","Type":"ContainerStarted","Data":"7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6"}
May 25 12:02:41.934709 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:41.934233    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" event={"ID":"3f8fbc35-5e88-4229-af1e-64c5dedd62f3","Type":"ContainerStarted","Data":"ec2f2aa609ca8de46f534d10e361c883ab36446a133eee5366c0a52a5c97ca19"}
May 25 12:02:41.934709 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:41.934417    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:41.935707 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:41.935679    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" event={"ID":"36388f0e-926b-4e51-8018-93324ff2b456","Type":"ContainerStarted","Data":"e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664"}
May 25 12:02:41.935821 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:41.935723    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" event={"ID":"36388f0e-926b-4e51-8018-93324ff2b456","Type":"ContainerStarted","Data":"0468ac7b459f8f747094a9b3a726a69391f6ab2e44c34a6961b749391ddd219d"}
May 25 12:02:42.941439 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:42.941378    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" event={"ID":"3f8fbc35-5e88-4229-af1e-64c5dedd62f3","Type":"ContainerStarted","Data":"08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721"}
May 25 12:02:46.958870 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:46.958834    2578 generic.go:358] "Generic (PLEG): container finished" podID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerID="08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721" exitCode=0
May 25 12:02:46.959389 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:46.958912    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" event={"ID":"3f8fbc35-5e88-4229-af1e-64c5dedd62f3","Type":"ContainerDied","Data":"08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721"}
May 25 12:02:46.960261 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:46.960238    2578 generic.go:358] "Generic (PLEG): container finished" podID="36388f0e-926b-4e51-8018-93324ff2b456" containerID="e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664" exitCode=0
May 25 12:02:46.960328 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:46.960310    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" event={"ID":"36388f0e-926b-4e51-8018-93324ff2b456","Type":"ContainerDied","Data":"e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664"}
May 25 12:02:47.966985 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:47.966923    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" event={"ID":"3f8fbc35-5e88-4229-af1e-64c5dedd62f3","Type":"ContainerStarted","Data":"2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79"}
May 25 12:02:47.968952 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:47.968920    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" event={"ID":"36388f0e-926b-4e51-8018-93324ff2b456","Type":"ContainerStarted","Data":"25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725"}
May 25 12:02:47.992899 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:47.992847    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podStartSLOduration=7.992826115 podStartE2EDuration="7.992826115s" podCreationTimestamp="2026-05-25 12:02:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:02:47.989400016 +0000 UTC m=+2427.638685011" watchObservedRunningTime="2026-05-25 12:02:47.992826115 +0000 UTC m=+2427.642111104"
May 25 12:02:48.012247 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:48.012178    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podStartSLOduration=8.012156442 podStartE2EDuration="8.012156442s" podCreationTimestamp="2026-05-25 12:02:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:02:48.009428706 +0000 UTC m=+2427.658713702" watchObservedRunningTime="2026-05-25 12:02:48.012156442 +0000 UTC m=+2427.661441453"
May 25 12:02:50.964253 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:50.964215    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:50.964761 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:50.964267    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:02:50.965951 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:50.965921    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:02:50.974816 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:50.974781    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:50.974945 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:50.974831    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:02:50.976284 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:02:50.976246    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:03:00.964911 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:00.964863    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:03:00.975018 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:00.974982    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:03:00.982326 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:00.982295    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:03:01.027522 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.027472    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="llm-d-routing-sidecar" containerID="cri-o://7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5" gracePeriod=2
May 25 12:03:01.318938 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.318912    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 12:03:01.354145 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.354113    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x_ff253078-cbf6-4087-b706-0abfd2da87f6/main/0.log"
May 25 12:03:01.355076 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.355051    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 12:03:01.463605 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.463559    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-model-cache\") pod \"ff253078-cbf6-4087-b706-0abfd2da87f6\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") "
May 25 12:03:01.463796 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.463613    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-kserve-provision-location\") pod \"ff253078-cbf6-4087-b706-0abfd2da87f6\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") "
May 25 12:03:01.463796 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.463662    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-model-cache\") pod \"f059415b-e1a6-4811-8d31-470b7703d996\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") "
May 25 12:03:01.463796 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.463696    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh2m7\" (UniqueName: \"kubernetes.io/projected/f059415b-e1a6-4811-8d31-470b7703d996-kube-api-access-sh2m7\") pod \"f059415b-e1a6-4811-8d31-470b7703d996\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") "
May 25 12:03:01.463796 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.463728    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-dshm\") pod \"f059415b-e1a6-4811-8d31-470b7703d996\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") "
May 25 12:03:01.464010 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.463882    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-dshm\") pod \"ff253078-cbf6-4087-b706-0abfd2da87f6\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") "
May 25 12:03:01.464010 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.463914    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff253078-cbf6-4087-b706-0abfd2da87f6" (UID: "ff253078-cbf6-4087-b706-0abfd2da87f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:03:01.464010 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.463988    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-model-cache" (OuterVolumeSpecName: "model-cache") pod "f059415b-e1a6-4811-8d31-470b7703d996" (UID: "f059415b-e1a6-4811-8d31-470b7703d996"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:03:01.464810 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.464214    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9swm\" (UniqueName: \"kubernetes.io/projected/ff253078-cbf6-4087-b706-0abfd2da87f6-kube-api-access-b9swm\") pod \"ff253078-cbf6-4087-b706-0abfd2da87f6\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") "
May 25 12:03:01.464810 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.464286    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f059415b-e1a6-4811-8d31-470b7703d996-tls-certs\") pod \"f059415b-e1a6-4811-8d31-470b7703d996\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") "
May 25 12:03:01.464810 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.464328    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-home\") pod \"ff253078-cbf6-4087-b706-0abfd2da87f6\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") "
May 25 12:03:01.464810 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.464368    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-tmp-dir\") pod \"ff253078-cbf6-4087-b706-0abfd2da87f6\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") "
May 25 12:03:01.464810 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.464406    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-kserve-provision-location\") pod \"f059415b-e1a6-4811-8d31-470b7703d996\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") "
May 25 12:03:01.464810 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.464430    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff253078-cbf6-4087-b706-0abfd2da87f6-tls-certs\") pod \"ff253078-cbf6-4087-b706-0abfd2da87f6\" (UID: \"ff253078-cbf6-4087-b706-0abfd2da87f6\") "
May 25 12:03:01.464810 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.464474    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-home\") pod \"f059415b-e1a6-4811-8d31-470b7703d996\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") "
May 25 12:03:01.464810 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.464505    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-tmp-dir\") pod \"f059415b-e1a6-4811-8d31-470b7703d996\" (UID: \"f059415b-e1a6-4811-8d31-470b7703d996\") "
May 25 12:03:01.465367 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.465298    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-home" (OuterVolumeSpecName: "home") pod "ff253078-cbf6-4087-b706-0abfd2da87f6" (UID: "ff253078-cbf6-4087-b706-0abfd2da87f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:03:01.465914 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.465626    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.465914 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.465653    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.465914 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.465667    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.468982 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.468927    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-home" (OuterVolumeSpecName: "home") pod "f059415b-e1a6-4811-8d31-470b7703d996" (UID: "f059415b-e1a6-4811-8d31-470b7703d996"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:03:01.468982 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.468953    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-dshm" (OuterVolumeSpecName: "dshm") pod "f059415b-e1a6-4811-8d31-470b7703d996" (UID: "f059415b-e1a6-4811-8d31-470b7703d996"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:03:01.469320 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.469276    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f059415b-e1a6-4811-8d31-470b7703d996-kube-api-access-sh2m7" (OuterVolumeSpecName: "kube-api-access-sh2m7") pod "f059415b-e1a6-4811-8d31-470b7703d996" (UID: "f059415b-e1a6-4811-8d31-470b7703d996"). InnerVolumeSpecName "kube-api-access-sh2m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:03:01.469861 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.469825    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-dshm" (OuterVolumeSpecName: "dshm") pod "ff253078-cbf6-4087-b706-0abfd2da87f6" (UID: "ff253078-cbf6-4087-b706-0abfd2da87f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:03:01.470077 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.470047    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f059415b-e1a6-4811-8d31-470b7703d996-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f059415b-e1a6-4811-8d31-470b7703d996" (UID: "f059415b-e1a6-4811-8d31-470b7703d996"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:03:01.470408 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.470377    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff253078-cbf6-4087-b706-0abfd2da87f6-kube-api-access-b9swm" (OuterVolumeSpecName: "kube-api-access-b9swm") pod "ff253078-cbf6-4087-b706-0abfd2da87f6" (UID: "ff253078-cbf6-4087-b706-0abfd2da87f6"). InnerVolumeSpecName "kube-api-access-b9swm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:03:01.470408 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.470388    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff253078-cbf6-4087-b706-0abfd2da87f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff253078-cbf6-4087-b706-0abfd2da87f6" (UID: "ff253078-cbf6-4087-b706-0abfd2da87f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:03:01.486745 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.486598    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff253078-cbf6-4087-b706-0abfd2da87f6" (UID: "ff253078-cbf6-4087-b706-0abfd2da87f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:03:01.487554 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.487526    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f059415b-e1a6-4811-8d31-470b7703d996" (UID: "f059415b-e1a6-4811-8d31-470b7703d996"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:03:01.526852 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.526799    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f059415b-e1a6-4811-8d31-470b7703d996" (UID: "f059415b-e1a6-4811-8d31-470b7703d996"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:03:01.532564 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.532504    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff253078-cbf6-4087-b706-0abfd2da87f6" (UID: "ff253078-cbf6-4087-b706-0abfd2da87f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:03:01.567000 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.566951    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.567000 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.566985    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh2m7\" (UniqueName: \"kubernetes.io/projected/f059415b-e1a6-4811-8d31-470b7703d996-kube-api-access-sh2m7\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.567000 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.566996    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.567000 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.567005    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.567000 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.567014    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9swm\" (UniqueName: \"kubernetes.io/projected/ff253078-cbf6-4087-b706-0abfd2da87f6-kube-api-access-b9swm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.567409 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.567023    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f059415b-e1a6-4811-8d31-470b7703d996-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.567409 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.567032    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff253078-cbf6-4087-b706-0abfd2da87f6-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.567409 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.567040    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.567409 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.567048    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff253078-cbf6-4087-b706-0abfd2da87f6-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.567409 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.567056    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:01.567409 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:01.567063    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f059415b-e1a6-4811-8d31-470b7703d996-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:03:02.030642 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.030599    2578 generic.go:358] "Generic (PLEG): container finished" podID="f059415b-e1a6-4811-8d31-470b7703d996" containerID="3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d" exitCode=137
May 25 12:03:02.031136 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.030668    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"
May 25 12:03:02.031136 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.030684    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" event={"ID":"f059415b-e1a6-4811-8d31-470b7703d996","Type":"ContainerDied","Data":"3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d"}
May 25 12:03:02.031136 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.030728    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b" event={"ID":"f059415b-e1a6-4811-8d31-470b7703d996","Type":"ContainerDied","Data":"5e7ce4fd38129f7b1dd307338177a85b7fd475969eaf6ff0de9e502f2751eb1c"}
May 25 12:03:02.031136 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.030752    2578 scope.go:117] "RemoveContainer" containerID="3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d"
May 25 12:03:02.032336 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.032319    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x_ff253078-cbf6-4087-b706-0abfd2da87f6/main/0.log"
May 25 12:03:02.033150 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.033126    2578 generic.go:358] "Generic (PLEG): container finished" podID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerID="706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e" exitCode=137
May 25 12:03:02.033150 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.033152    2578 generic.go:358] "Generic (PLEG): container finished" podID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerID="7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5" exitCode=0
May 25 12:03:02.033313 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.033203    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" event={"ID":"ff253078-cbf6-4087-b706-0abfd2da87f6","Type":"ContainerDied","Data":"706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e"}
May 25 12:03:02.033313 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.033231    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" event={"ID":"ff253078-cbf6-4087-b706-0abfd2da87f6","Type":"ContainerDied","Data":"7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5"}
May 25 12:03:02.033313 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.033249    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x" event={"ID":"ff253078-cbf6-4087-b706-0abfd2da87f6","Type":"ContainerDied","Data":"7a88dfb34ce4e973c84fba0a4b7afae9352e5b35b875f1ad75668b2016bc1c2f"}
May 25 12:03:02.033313 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.033271    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"
May 25 12:03:02.049815 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.049793    2578 scope.go:117] "RemoveContainer" containerID="3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f"
May 25 12:03:02.061598 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.061573    2578 scope.go:117] "RemoveContainer" containerID="3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d"
May 25 12:03:02.061956 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:03:02.061929    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d\": container with ID starting with 3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d not found: ID does not exist" containerID="3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d"
May 25 12:03:02.062044 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.061972    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d"} err="failed to get container status \"3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d\": rpc error: code = NotFound desc = could not find container \"3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d\": container with ID starting with 3fc3dcda66279710694696955f916678df2ed1be7fa6746369606a49bd884c0d not found: ID does not exist"
May 25 12:03:02.062044 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.061999    2578 scope.go:117] "RemoveContainer" containerID="3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f"
May 25 12:03:02.062336 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:03:02.062304    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f\": container with ID starting with 3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f not found: ID does not exist" containerID="3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f"
May 25 12:03:02.062429 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.062342    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f"} err="failed to get container status \"3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f\": rpc error: code = NotFound desc = could not find container \"3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f\": container with ID starting with 3a4c3c74bee342d056cf62c0ba2a88abcbcd2e58d2f2a24e28bf4bb270258c2f not found: ID does not exist"
May 25 12:03:02.062429 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.062366    2578 scope.go:117] "RemoveContainer" containerID="706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e"
May 25 12:03:02.074678 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.074637    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"]
May 25 12:03:02.078442 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.078415    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-6645b5cfbd5826x"]
May 25 12:03:02.083304 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.083278    2578 scope.go:117] "RemoveContainer" containerID="865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792"
May 25 12:03:02.091090 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.091047    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"]
May 25 12:03:02.095989 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.095958    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-678rh8b"]
May 25 12:03:02.157634 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.157598    2578 scope.go:117] "RemoveContainer" containerID="7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5"
May 25 12:03:02.166370 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.166339    2578 scope.go:117] "RemoveContainer" containerID="706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e"
May 25 12:03:02.166710 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:03:02.166684    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e\": container with ID starting with 706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e not found: ID does not exist" containerID="706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e"
May 25 12:03:02.166848 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.166716    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e"} err="failed to get container status \"706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e\": rpc error: code = NotFound desc = could not find container \"706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e\": container with ID starting with 706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e not found: ID does not exist"
May 25 12:03:02.166848 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.166739    2578 scope.go:117] "RemoveContainer" containerID="865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792"
May 25 12:03:02.166979 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:03:02.166948    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792\": container with ID starting with 865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792 not found: ID does not exist" containerID="865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792"
May 25 12:03:02.167047 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.166970    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792"} err="failed to get container status \"865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792\": rpc error: code = NotFound desc = could not find container \"865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792\": container with ID starting with 865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792 not found: ID does not exist"
May 25 12:03:02.167047 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.166993    2578 scope.go:117] "RemoveContainer" containerID="7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5"
May 25 12:03:02.167239 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:03:02.167220    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5\": container with ID starting with 7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5 not found: ID does not exist" containerID="7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5"
May 25 12:03:02.167315 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.167253    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5"} err="failed to get container status \"7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5\": rpc error: code = NotFound desc = could not find container \"7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5\": container with ID starting with 7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5 not found: ID does not exist"
May 25 12:03:02.167315 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.167270    2578 scope.go:117] "RemoveContainer" containerID="706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e"
May 25 12:03:02.167532 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.167512    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e"} err="failed to get container status \"706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e\": rpc error: code = NotFound desc = could not find container \"706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e\": container with ID starting with 706c190b63c36ddf4d994b46083119d2ab514e2ed783b569118ea5eccce7a37e not found: ID does not exist"
May 25 12:03:02.167532 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.167531    2578 scope.go:117] "RemoveContainer" containerID="865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792"
May 25 12:03:02.167802 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.167778    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792"} err="failed to get container status \"865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792\": rpc error: code = NotFound desc = could not find container \"865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792\": container with ID starting with 865472b7c315a27449e66673e3e1d3c184b334d67b4b0956f8831946c14b4792 not found: ID does not exist"
May 25 12:03:02.167802 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.167802    2578 scope.go:117] "RemoveContainer" containerID="7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5"
May 25 12:03:02.168003 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.167986    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5"} err="failed to get container status \"7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5\": rpc error: code = NotFound desc = could not find container \"7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5\": container with ID starting with 7eb03da01ae9b3f5a787ab8732655d5dca976d93f9b88bb702a0b524cb95f1a5 not found: ID does not exist"
May 25 12:03:02.935618 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.935581    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f059415b-e1a6-4811-8d31-470b7703d996" path="/var/lib/kubelet/pods/f059415b-e1a6-4811-8d31-470b7703d996/volumes"
May 25 12:03:02.936030 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:02.936012    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" path="/var/lib/kubelet/pods/ff253078-cbf6-4087-b706-0abfd2da87f6/volumes"
May 25 12:03:10.964867 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:10.964798    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:03:10.974863 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:10.974817    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:03:20.965005 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:20.964944    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:03:20.975036 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:20.974990    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:03:30.964876 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:30.964814    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:03:30.975323 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:30.975273    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:03:40.965413 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:40.965358    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:03:40.975583 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:40.975550    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:03:50.964700 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:50.964642    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:03:50.974936 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:03:50.974889    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:04:00.964898 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:00.964846    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:04:00.974917 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:00.974873    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:04:10.964761 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:10.964654    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:04:10.976270 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:10.976234    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:04:20.964585 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:20.964533    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:04:20.975640 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:20.975591    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:04:30.964986 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:30.964921    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:04:30.975401 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:30.975365    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:04:40.965439 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:40.965389    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:04:40.975152 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:40.975111    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:04:50.964869 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:50.964813    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:04:50.975803 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:04:50.975764    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:05:00.964723 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:00.964675    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 25 12:05:00.975442 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:00.975398    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 25 12:05:10.975093 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:10.975055    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:05:10.990080 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:10.990051    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:05:10.992083 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:10.992055    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:05:10.998238 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:10.998216    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:05:19.583782 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:19.583737    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"]
May 25 12:05:19.584271 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:19.584174    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main" containerID="cri-o://2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79" gracePeriod=30
May 25 12:05:19.586106 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:19.586081    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"]
May 25 12:05:19.586355 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:19.586325    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main" containerID="cri-o://25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725" gracePeriod=30
May 25 12:05:25.606557 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:25.606438    2578 generic.go:358] "Generic (PLEG): container finished" podID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerID="80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6" exitCode=0
May 25 12:05:25.606557 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:25.606524    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" event={"ID":"0ada20ec-3961-4c62-9c41-55d7a22a519d","Type":"ContainerDied","Data":"80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6"}
May 25 12:05:25.607879 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:25.607862    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:05:26.612054 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:26.612014    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" event={"ID":"0ada20ec-3961-4c62-9c41-55d7a22a519d","Type":"ContainerStarted","Data":"37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707"}
May 25 12:05:26.638221 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:26.638161    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podStartSLOduration=199.638139423 podStartE2EDuration="3m19.638139423s" podCreationTimestamp="2026-05-25 12:02:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:05:26.635300531 +0000 UTC m=+2586.284585538" watchObservedRunningTime="2026-05-25 12:05:26.638139423 +0000 UTC m=+2586.287424603"
May 25 12:05:27.737918 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:27.737870    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:05:27.737918 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:27.737927    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:05:27.739790 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:27.739751    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 25 12:05:37.737864 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:37.737813    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 25 12:05:47.737719 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:47.737677    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 25 12:05:49.584399 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.584351    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="llm-d-routing-sidecar" containerID="cri-o://7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6" gracePeriod=2
May 25 12:05:49.703049 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.703019    2578 generic.go:358] "Generic (PLEG): container finished" podID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerID="7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6" exitCode=0
May 25 12:05:49.703343 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.703095    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" event={"ID":"3f8fbc35-5e88-4229-af1e-64c5dedd62f3","Type":"ContainerDied","Data":"7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6"}
May 25 12:05:49.883226 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.883199    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv_3f8fbc35-5e88-4229-af1e-64c5dedd62f3/main/0.log"
May 25 12:05:49.884069 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.884047    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:05:49.887108 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.887086    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:05:49.975426 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975386    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tls-certs\") pod \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") "
May 25 12:05:49.975638 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975440    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t55vf\" (UniqueName: \"kubernetes.io/projected/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kube-api-access-t55vf\") pod \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") "
May 25 12:05:49.975638 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975518    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tmp-dir\") pod \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") "
May 25 12:05:49.975638 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975547    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-dshm\") pod \"36388f0e-926b-4e51-8018-93324ff2b456\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") "
May 25 12:05:49.975638 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975577    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kserve-provision-location\") pod \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") "
May 25 12:05:49.975638 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975602    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36388f0e-926b-4e51-8018-93324ff2b456-tls-certs\") pod \"36388f0e-926b-4e51-8018-93324ff2b456\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") "
May 25 12:05:49.975908 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975653    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-home\") pod \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") "
May 25 12:05:49.975908 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975682    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-model-cache\") pod \"36388f0e-926b-4e51-8018-93324ff2b456\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") "
May 25 12:05:49.975908 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975734    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-home\") pod \"36388f0e-926b-4e51-8018-93324ff2b456\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") "
May 25 12:05:49.975908 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975765    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqfvv\" (UniqueName: \"kubernetes.io/projected/36388f0e-926b-4e51-8018-93324ff2b456-kube-api-access-cqfvv\") pod \"36388f0e-926b-4e51-8018-93324ff2b456\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") "
May 25 12:05:49.975908 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975791    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-dshm\") pod \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") "
May 25 12:05:49.975908 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975816    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-model-cache\") pod \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\" (UID: \"3f8fbc35-5e88-4229-af1e-64c5dedd62f3\") "
May 25 12:05:49.975908 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975839    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-tmp-dir\") pod \"36388f0e-926b-4e51-8018-93324ff2b456\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") "
May 25 12:05:49.975908 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.975865    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-kserve-provision-location\") pod \"36388f0e-926b-4e51-8018-93324ff2b456\" (UID: \"36388f0e-926b-4e51-8018-93324ff2b456\") "
May 25 12:05:49.979925 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.976589    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f8fbc35-5e88-4229-af1e-64c5dedd62f3" (UID: "3f8fbc35-5e88-4229-af1e-64c5dedd62f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:05:49.979925 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.978784    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-model-cache" (OuterVolumeSpecName: "model-cache") pod "36388f0e-926b-4e51-8018-93324ff2b456" (UID: "36388f0e-926b-4e51-8018-93324ff2b456"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:05:49.979925 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.979329    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-home" (OuterVolumeSpecName: "home") pod "3f8fbc35-5e88-4229-af1e-64c5dedd62f3" (UID: "3f8fbc35-5e88-4229-af1e-64c5dedd62f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:05:49.979925 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.979388    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-home" (OuterVolumeSpecName: "home") pod "36388f0e-926b-4e51-8018-93324ff2b456" (UID: "36388f0e-926b-4e51-8018-93324ff2b456"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:05:49.979925 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.979497    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kube-api-access-t55vf" (OuterVolumeSpecName: "kube-api-access-t55vf") pod "3f8fbc35-5e88-4229-af1e-64c5dedd62f3" (UID: "3f8fbc35-5e88-4229-af1e-64c5dedd62f3"). InnerVolumeSpecName "kube-api-access-t55vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:05:49.980314 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.979979    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36388f0e-926b-4e51-8018-93324ff2b456-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36388f0e-926b-4e51-8018-93324ff2b456" (UID: "36388f0e-926b-4e51-8018-93324ff2b456"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:05:49.980997 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.980946    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f8fbc35-5e88-4229-af1e-64c5dedd62f3" (UID: "3f8fbc35-5e88-4229-af1e-64c5dedd62f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:05:49.981123 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.981048    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36388f0e-926b-4e51-8018-93324ff2b456-kube-api-access-cqfvv" (OuterVolumeSpecName: "kube-api-access-cqfvv") pod "36388f0e-926b-4e51-8018-93324ff2b456" (UID: "36388f0e-926b-4e51-8018-93324ff2b456"). InnerVolumeSpecName "kube-api-access-cqfvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:05:49.981278 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.981257    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-dshm" (OuterVolumeSpecName: "dshm") pod "36388f0e-926b-4e51-8018-93324ff2b456" (UID: "36388f0e-926b-4e51-8018-93324ff2b456"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:05:49.981420 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.981404    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-dshm" (OuterVolumeSpecName: "dshm") pod "3f8fbc35-5e88-4229-af1e-64c5dedd62f3" (UID: "3f8fbc35-5e88-4229-af1e-64c5dedd62f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:05:49.993063 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.993033    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f8fbc35-5e88-4229-af1e-64c5dedd62f3" (UID: "3f8fbc35-5e88-4229-af1e-64c5dedd62f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:05:49.997967 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:49.997933    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36388f0e-926b-4e51-8018-93324ff2b456" (UID: "36388f0e-926b-4e51-8018-93324ff2b456"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:05:50.042873 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.042828    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f8fbc35-5e88-4229-af1e-64c5dedd62f3" (UID: "3f8fbc35-5e88-4229-af1e-64c5dedd62f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:05:50.048286 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.048256    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36388f0e-926b-4e51-8018-93324ff2b456" (UID: "36388f0e-926b-4e51-8018-93324ff2b456"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:05:50.076787 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076754    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.076787 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076785    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.076787 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076796    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqfvv\" (UniqueName: \"kubernetes.io/projected/36388f0e-926b-4e51-8018-93324ff2b456-kube-api-access-cqfvv\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.077017 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076807    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.077017 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076816    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.077017 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076825    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.077017 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076834    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.077017 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076842    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.077017 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076851    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t55vf\" (UniqueName: \"kubernetes.io/projected/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kube-api-access-t55vf\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.077017 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076859    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.077017 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076866    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36388f0e-926b-4e51-8018-93324ff2b456-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.077017 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076874    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.077017 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076882    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36388f0e-926b-4e51-8018-93324ff2b456-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.077017 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.076890    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f8fbc35-5e88-4229-af1e-64c5dedd62f3-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:05:50.708777 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.708734    2578 generic.go:358] "Generic (PLEG): container finished" podID="36388f0e-926b-4e51-8018-93324ff2b456" containerID="25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725" exitCode=137
May 25 12:05:50.709243 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.708787    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" event={"ID":"36388f0e-926b-4e51-8018-93324ff2b456","Type":"ContainerDied","Data":"25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725"}
May 25 12:05:50.709243 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.708818    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"
May 25 12:05:50.709243 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.708833    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62" event={"ID":"36388f0e-926b-4e51-8018-93324ff2b456","Type":"ContainerDied","Data":"0468ac7b459f8f747094a9b3a726a69391f6ab2e44c34a6961b749391ddd219d"}
May 25 12:05:50.709243 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.708851    2578 scope.go:117] "RemoveContainer" containerID="25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725"
May 25 12:05:50.710361 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.710344    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv_3f8fbc35-5e88-4229-af1e-64c5dedd62f3/main/0.log"
May 25 12:05:50.711073 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.711047    2578 generic.go:358] "Generic (PLEG): container finished" podID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerID="2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79" exitCode=137
May 25 12:05:50.711197 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.711087    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" event={"ID":"3f8fbc35-5e88-4229-af1e-64c5dedd62f3","Type":"ContainerDied","Data":"2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79"}
May 25 12:05:50.711197 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.711113    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv" event={"ID":"3f8fbc35-5e88-4229-af1e-64c5dedd62f3","Type":"ContainerDied","Data":"ec2f2aa609ca8de46f534d10e361c883ab36446a133eee5366c0a52a5c97ca19"}
May 25 12:05:50.711197 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.711125    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"
May 25 12:05:50.726834 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.726805    2578 scope.go:117] "RemoveContainer" containerID="e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664"
May 25 12:05:50.737653 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.737629    2578 scope.go:117] "RemoveContainer" containerID="25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725"
May 25 12:05:50.737978 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:05:50.737959    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725\": container with ID starting with 25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725 not found: ID does not exist" containerID="25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725"
May 25 12:05:50.738025 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.737989    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725"} err="failed to get container status \"25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725\": rpc error: code = NotFound desc = could not find container \"25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725\": container with ID starting with 25f365656610b19a28fa00fefdbd9a3a30553aa82ed649b7b5c9b6f34620f725 not found: ID does not exist"
May 25 12:05:50.738025 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.738010    2578 scope.go:117] "RemoveContainer" containerID="e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664"
May 25 12:05:50.738304 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:05:50.738285    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664\": container with ID starting with e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664 not found: ID does not exist" containerID="e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664"
May 25 12:05:50.738378 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.738310    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664"} err="failed to get container status \"e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664\": rpc error: code = NotFound desc = could not find container \"e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664\": container with ID starting with e7d38e0311c0325a6eea1b45ddbb68379e59cbbdff90885737f48e596cb92664 not found: ID does not exist"
May 25 12:05:50.738378 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.738335    2578 scope.go:117] "RemoveContainer" containerID="2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79"
May 25 12:05:50.742656 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.742627    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"]
May 25 12:05:50.747175 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.747149    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6dc4cccd6-2dc62"]
May 25 12:05:50.749177 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.749157    2578 scope.go:117] "RemoveContainer" containerID="08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721"
May 25 12:05:50.759536 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.759505    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"]
May 25 12:05:50.761516 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.761479    2578 scope.go:117] "RemoveContainer" containerID="7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6"
May 25 12:05:50.763288 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.763262    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-54b44b7855-gbrkv"]
May 25 12:05:50.769995 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.769972    2578 scope.go:117] "RemoveContainer" containerID="2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79"
May 25 12:05:50.770336 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:05:50.770309    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79\": container with ID starting with 2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79 not found: ID does not exist" containerID="2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79"
May 25 12:05:50.770420 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.770351    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79"} err="failed to get container status \"2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79\": rpc error: code = NotFound desc = could not find container \"2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79\": container with ID starting with 2f33acb7a41928727258a0b255f3608a567b904f033204e47227a9b1a7980b79 not found: ID does not exist"
May 25 12:05:50.770420 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.770380    2578 scope.go:117] "RemoveContainer" containerID="08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721"
May 25 12:05:50.770694 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:05:50.770673    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721\": container with ID starting with 08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721 not found: ID does not exist" containerID="08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721"
May 25 12:05:50.770742 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.770706    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721"} err="failed to get container status \"08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721\": rpc error: code = NotFound desc = could not find container \"08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721\": container with ID starting with 08081c5be1df2da8fcd4dbf6c68334f15fa9e8faec4af6bf4d35b78b2df69721 not found: ID does not exist"
May 25 12:05:50.770742 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.770730    2578 scope.go:117] "RemoveContainer" containerID="7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6"
May 25 12:05:50.770987 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:05:50.770969    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6\": container with ID starting with 7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6 not found: ID does not exist" containerID="7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6"
May 25 12:05:50.771049 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.770992    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6"} err="failed to get container status \"7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6\": rpc error: code = NotFound desc = could not find container \"7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6\": container with ID starting with 7e8d060d773c2f3fbb153da0b3018e8438492da7bbb72d6176201dfec3bcf5d6 not found: ID does not exist"
May 25 12:05:50.935074 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.934991    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36388f0e-926b-4e51-8018-93324ff2b456" path="/var/lib/kubelet/pods/36388f0e-926b-4e51-8018-93324ff2b456/volumes"
May 25 12:05:50.935668 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:50.935648    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" path="/var/lib/kubelet/pods/3f8fbc35-5e88-4229-af1e-64c5dedd62f3/volumes"
May 25 12:05:57.738088 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:05:57.738034    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 25 12:06:07.738085 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:06:07.738023    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 25 12:06:17.737923 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:06:17.737862    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 25 12:06:27.738025 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:06:27.737969    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 25 12:06:37.737769 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:06:37.737725    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 25 12:06:47.738197 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:06:47.738149    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 25 12:06:57.737681 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:06:57.737631    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 25 12:07:07.747472 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:07.747434    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:07:07.755345 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:07.755319    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:07:12.407815 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:12.407782    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"]
May 25 12:07:12.408191 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:12.408066    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main" containerID="cri-o://37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707" gracePeriod=30
May 25 12:07:21.131344 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:21.131319    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 12:07:21.138008 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:21.137988    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 12:07:26.530951 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.530911    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531244    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="llm-d-routing-sidecar"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531258    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="llm-d-routing-sidecar"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531269    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="llm-d-routing-sidecar"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531275    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="llm-d-routing-sidecar"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531286    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="storage-initializer"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531292    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="storage-initializer"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531301    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531306    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531314    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531320    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531327    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531332    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main"
May 25 12:07:26.531337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531342    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="storage-initializer"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531347    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="storage-initializer"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531354    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="storage-initializer"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531359    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="storage-initializer"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531367    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531372    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531378    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="storage-initializer"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531382    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="storage-initializer"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531427    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="main"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531436    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="f059415b-e1a6-4811-8d31-470b7703d996" containerName="main"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531441    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="3f8fbc35-5e88-4229-af1e-64c5dedd62f3" containerName="llm-d-routing-sidecar"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531448    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="llm-d-routing-sidecar"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531471    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="ff253078-cbf6-4087-b706-0abfd2da87f6" containerName="main"
May 25 12:07:26.531765 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.531479    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="36388f0e-926b-4e51-8018-93324ff2b456" containerName="main"
May 25 12:07:26.534547 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.534528    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.537240 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.537218    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-zzmgl\""
May 25 12:07:26.537240 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.537228    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 25 12:07:26.544188 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.544161    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 12:07:26.699432 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.699401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4t5\" (UniqueName: \"kubernetes.io/projected/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kube-api-access-kv4t5\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.699432 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.699444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.699730 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.699526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.699730 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.699607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.699730 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.699631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.699730 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.699705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.699921 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.699787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.801231 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.801141    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.801231 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.801206    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.801480 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.801231    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.801480 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.801262    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.801480 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.801304    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.801480 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.801330    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4t5\" (UniqueName: \"kubernetes.io/projected/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kube-api-access-kv4t5\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.801480 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.801447    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.801759 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.801634    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.801820 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.801763    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.801820 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.801794    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.801820 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.801799    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.803819 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.803795    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.803943 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.803916    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.814324 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.814291    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4t5\" (UniqueName: \"kubernetes.io/projected/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kube-api-access-kv4t5\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.847439 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.847399    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:07:26.989019 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:26.988974    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 12:07:26.991031 ip-10-0-133-58 kubenswrapper[2578]: W0525 12:07:26.991001    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a8975d_9fa8_4687_95d2_7b4bcbeaed17.slice/crio-366b91c995c6a10cc61dee921f388c0acb91408949c9d1cb845cbcf4d001c14b WatchSource:0}: Error finding container 366b91c995c6a10cc61dee921f388c0acb91408949c9d1cb845cbcf4d001c14b: Status 404 returned error can't find the container with id 366b91c995c6a10cc61dee921f388c0acb91408949c9d1cb845cbcf4d001c14b
May 25 12:07:27.081768 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:27.081731    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"27a8975d-9fa8-4687-95d2-7b4bcbeaed17","Type":"ContainerStarted","Data":"66b1976700d1d6971275e0f7dd96dbd99dd34bf76e61d2518adf4d8367412a8b"}
May 25 12:07:27.081768 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:27.081773    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"27a8975d-9fa8-4687-95d2-7b4bcbeaed17","Type":"ContainerStarted","Data":"366b91c995c6a10cc61dee921f388c0acb91408949c9d1cb845cbcf4d001c14b"}
May 25 12:07:42.669347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.669282    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w_0ada20ec-3961-4c62-9c41-55d7a22a519d/main/0.log"
May 25 12:07:42.669748 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.669731    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:07:42.720424 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.720393    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-tmp-dir\") pod \"0ada20ec-3961-4c62-9c41-55d7a22a519d\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") "
May 25 12:07:42.720424 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.720430    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-home\") pod \"0ada20ec-3961-4c62-9c41-55d7a22a519d\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") "
May 25 12:07:42.720692 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.720482    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ada20ec-3961-4c62-9c41-55d7a22a519d-tls-certs\") pod \"0ada20ec-3961-4c62-9c41-55d7a22a519d\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") "
May 25 12:07:42.720692 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.720506    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-model-cache\") pod \"0ada20ec-3961-4c62-9c41-55d7a22a519d\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") "
May 25 12:07:42.720692 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.720528    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-dshm\") pod \"0ada20ec-3961-4c62-9c41-55d7a22a519d\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") "
May 25 12:07:42.720692 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.720552    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-kserve-provision-location\") pod \"0ada20ec-3961-4c62-9c41-55d7a22a519d\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") "
May 25 12:07:42.720692 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.720588    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7glx\" (UniqueName: \"kubernetes.io/projected/0ada20ec-3961-4c62-9c41-55d7a22a519d-kube-api-access-l7glx\") pod \"0ada20ec-3961-4c62-9c41-55d7a22a519d\" (UID: \"0ada20ec-3961-4c62-9c41-55d7a22a519d\") "
May 25 12:07:42.720958 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.720832    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ada20ec-3961-4c62-9c41-55d7a22a519d" (UID: "0ada20ec-3961-4c62-9c41-55d7a22a519d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:07:42.721261 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.721189    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-home" (OuterVolumeSpecName: "home") pod "0ada20ec-3961-4c62-9c41-55d7a22a519d" (UID: "0ada20ec-3961-4c62-9c41-55d7a22a519d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:07:42.722857 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.722833    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ada20ec-3961-4c62-9c41-55d7a22a519d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ada20ec-3961-4c62-9c41-55d7a22a519d" (UID: "0ada20ec-3961-4c62-9c41-55d7a22a519d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:07:42.723268 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.723248    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-dshm" (OuterVolumeSpecName: "dshm") pod "0ada20ec-3961-4c62-9c41-55d7a22a519d" (UID: "0ada20ec-3961-4c62-9c41-55d7a22a519d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:07:42.723492 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.723437    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ada20ec-3961-4c62-9c41-55d7a22a519d-kube-api-access-l7glx" (OuterVolumeSpecName: "kube-api-access-l7glx") pod "0ada20ec-3961-4c62-9c41-55d7a22a519d" (UID: "0ada20ec-3961-4c62-9c41-55d7a22a519d"). InnerVolumeSpecName "kube-api-access-l7glx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:07:42.741697 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.741658    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ada20ec-3961-4c62-9c41-55d7a22a519d" (UID: "0ada20ec-3961-4c62-9c41-55d7a22a519d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:07:42.785875 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.785832    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ada20ec-3961-4c62-9c41-55d7a22a519d" (UID: "0ada20ec-3961-4c62-9c41-55d7a22a519d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:07:42.821878 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.821843    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:07:42.821878 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.821872    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:07:42.821878 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.821881    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ada20ec-3961-4c62-9c41-55d7a22a519d-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:07:42.822093 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.821891    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:07:42.822093 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.821899    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:07:42.822093 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.821909    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ada20ec-3961-4c62-9c41-55d7a22a519d-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:07:42.822093 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:42.821918    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7glx\" (UniqueName: \"kubernetes.io/projected/0ada20ec-3961-4c62-9c41-55d7a22a519d-kube-api-access-l7glx\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:07:43.141635 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.141599    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w_0ada20ec-3961-4c62-9c41-55d7a22a519d/main/0.log"
May 25 12:07:43.141950 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.141926    2578 generic.go:358] "Generic (PLEG): container finished" podID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerID="37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707" exitCode=137
May 25 12:07:43.142059 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.142018    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"
May 25 12:07:43.142105 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.142017    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" event={"ID":"0ada20ec-3961-4c62-9c41-55d7a22a519d","Type":"ContainerDied","Data":"37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707"}
May 25 12:07:43.142148 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.142132    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w" event={"ID":"0ada20ec-3961-4c62-9c41-55d7a22a519d","Type":"ContainerDied","Data":"fe2b84664f186624db1870f6d4d1e1c80e14d83f1d35a0818df6e32539564034"}
May 25 12:07:43.142194 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.142153    2578 scope.go:117] "RemoveContainer" containerID="37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707"
May 25 12:07:43.151034 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.151013    2578 scope.go:117] "RemoveContainer" containerID="80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6"
May 25 12:07:43.167655 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.167626    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"]
May 25 12:07:43.177636 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.177608    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7695869749csl4w"]
May 25 12:07:43.217669 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.217649    2578 scope.go:117] "RemoveContainer" containerID="37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707"
May 25 12:07:43.218014 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:07:43.217983    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707\": container with ID starting with 37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707 not found: ID does not exist" containerID="37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707"
May 25 12:07:43.218067 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.218014    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707"} err="failed to get container status \"37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707\": rpc error: code = NotFound desc = could not find container \"37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707\": container with ID starting with 37ea4df6c5b8b4bee0fb82262a8999bfb44b4ab2da018b2f04fe09008a771707 not found: ID does not exist"
May 25 12:07:43.218067 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.218035    2578 scope.go:117] "RemoveContainer" containerID="80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6"
May 25 12:07:43.218306 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:07:43.218285    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6\": container with ID starting with 80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6 not found: ID does not exist" containerID="80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6"
May 25 12:07:43.218376 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:43.218317    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6"} err="failed to get container status \"80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6\": rpc error: code = NotFound desc = could not find container \"80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6\": container with ID starting with 80984aabe0973660f0ebec0b066e053790dd6d9e14c622fb0769b568c9e826d6 not found: ID does not exist"
May 25 12:07:44.935408 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:07:44.935373    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" path="/var/lib/kubelet/pods/0ada20ec-3961-4c62-9c41-55d7a22a519d/volumes"
May 25 12:11:30.967323 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:11:30.967289    2578 generic.go:358] "Generic (PLEG): container finished" podID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerID="66b1976700d1d6971275e0f7dd96dbd99dd34bf76e61d2518adf4d8367412a8b" exitCode=0
May 25 12:11:30.967773 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:11:30.967363    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"27a8975d-9fa8-4687-95d2-7b4bcbeaed17","Type":"ContainerDied","Data":"66b1976700d1d6971275e0f7dd96dbd99dd34bf76e61d2518adf4d8367412a8b"}
May 25 12:11:30.968632 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:11:30.968614    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:11:31.972744 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:11:31.972711    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"27a8975d-9fa8-4687-95d2-7b4bcbeaed17","Type":"ContainerStarted","Data":"9e71f9a7e89804d6d3dfbe7d6942fb4b45b9ff51a480f645a0e5d2efa6e4dede"}
May 25 12:11:31.999326 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:11:31.999261    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=245.999240306 podStartE2EDuration="4m5.999240306s" podCreationTimestamp="2026-05-25 12:07:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:11:31.996737696 +0000 UTC m=+2951.646022693" watchObservedRunningTime="2026-05-25 12:11:31.999240306 +0000 UTC m=+2951.648525310"
May 25 12:11:36.848108 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:11:36.848060    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:11:36.849578 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:11:36.849551    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 25 12:11:46.848968 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:11:46.848873    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 25 12:11:56.848087 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:11:56.848045    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:11:56.848606 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:11:56.848399    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 25 12:12:06.848558 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:12:06.848504    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 25 12:12:16.848290 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:12:16.848246    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 25 12:12:21.163649 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:12:21.163605    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 12:12:21.170803 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:12:21.170779    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 12:12:26.848159 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:12:26.848117    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 25 12:12:36.848110 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:12:36.848061    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 25 12:12:46.848705 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:12:46.848651    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 25 12:12:56.848010 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:12:56.847963    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 25 12:13:06.857930 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:06.857898    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:13:06.865569 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:06.865544    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:13:14.644337 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:14.644249    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 12:13:14.644770 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:14.644572    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main" containerID="cri-o://9e71f9a7e89804d6d3dfbe7d6942fb4b45b9ff51a480f645a0e5d2efa6e4dede" gracePeriod=30
May 25 12:13:15.374064 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.374029    2578 generic.go:358] "Generic (PLEG): container finished" podID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerID="9e71f9a7e89804d6d3dfbe7d6942fb4b45b9ff51a480f645a0e5d2efa6e4dede" exitCode=0
May 25 12:13:15.374206 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.374093    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"27a8975d-9fa8-4687-95d2-7b4bcbeaed17","Type":"ContainerDied","Data":"9e71f9a7e89804d6d3dfbe7d6942fb4b45b9ff51a480f645a0e5d2efa6e4dede"}
May 25 12:13:15.374206 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.374127    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"27a8975d-9fa8-4687-95d2-7b4bcbeaed17","Type":"ContainerDied","Data":"366b91c995c6a10cc61dee921f388c0acb91408949c9d1cb845cbcf4d001c14b"}
May 25 12:13:15.374206 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.374137    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="366b91c995c6a10cc61dee921f388c0acb91408949c9d1cb845cbcf4d001c14b"
May 25 12:13:15.384676 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.384653    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:13:15.497225 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.497191    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-dshm\") pod \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") "
May 25 12:13:15.497421 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.497258    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kserve-provision-location\") pod \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") "
May 25 12:13:15.497421 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.497289    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tls-certs\") pod \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") "
May 25 12:13:15.497421 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.497315    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tmp-dir\") pod \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") "
May 25 12:13:15.497421 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.497349    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-model-cache\") pod \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") "
May 25 12:13:15.497421 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.497396    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv4t5\" (UniqueName: \"kubernetes.io/projected/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kube-api-access-kv4t5\") pod \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") "
May 25 12:13:15.497783 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.497438    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-home\") pod \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\" (UID: \"27a8975d-9fa8-4687-95d2-7b4bcbeaed17\") "
May 25 12:13:15.498380 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.498343    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-model-cache" (OuterVolumeSpecName: "model-cache") pod "27a8975d-9fa8-4687-95d2-7b4bcbeaed17" (UID: "27a8975d-9fa8-4687-95d2-7b4bcbeaed17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:13:15.498532 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.498401    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-home" (OuterVolumeSpecName: "home") pod "27a8975d-9fa8-4687-95d2-7b4bcbeaed17" (UID: "27a8975d-9fa8-4687-95d2-7b4bcbeaed17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:13:15.499717 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.499685    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27a8975d-9fa8-4687-95d2-7b4bcbeaed17" (UID: "27a8975d-9fa8-4687-95d2-7b4bcbeaed17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:13:15.499829 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.499744    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kube-api-access-kv4t5" (OuterVolumeSpecName: "kube-api-access-kv4t5") pod "27a8975d-9fa8-4687-95d2-7b4bcbeaed17" (UID: "27a8975d-9fa8-4687-95d2-7b4bcbeaed17"). InnerVolumeSpecName "kube-api-access-kv4t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:13:15.500410 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.500380    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-dshm" (OuterVolumeSpecName: "dshm") pod "27a8975d-9fa8-4687-95d2-7b4bcbeaed17" (UID: "27a8975d-9fa8-4687-95d2-7b4bcbeaed17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:13:15.510728 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.510704    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27a8975d-9fa8-4687-95d2-7b4bcbeaed17" (UID: "27a8975d-9fa8-4687-95d2-7b4bcbeaed17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:13:15.565080 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.565034    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27a8975d-9fa8-4687-95d2-7b4bcbeaed17" (UID: "27a8975d-9fa8-4687-95d2-7b4bcbeaed17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:13:15.598998 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.598966    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-home\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:13:15.598998 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.598998    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-dshm\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:13:15.599112 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.599008    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kserve-provision-location\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:13:15.599112 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.599018    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tls-certs\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:13:15.599112 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.599028    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-tmp-dir\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:13:15.599112 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.599036    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-model-cache\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:13:15.599112 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.599045    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv4t5\" (UniqueName: \"kubernetes.io/projected/27a8975d-9fa8-4687-95d2-7b4bcbeaed17-kube-api-access-kv4t5\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:13:15.804962 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.804878    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mlrfx/must-gather-cq7bg"]
May 25 12:13:15.805347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.805166    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="storage-initializer"
May 25 12:13:15.805347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.805177    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="storage-initializer"
May 25 12:13:15.805347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.805189    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="storage-initializer"
May 25 12:13:15.805347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.805195    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="storage-initializer"
May 25 12:13:15.805347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.805202    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main"
May 25 12:13:15.805347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.805208    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main"
May 25 12:13:15.805347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.805221    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main"
May 25 12:13:15.805347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.805226    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main"
May 25 12:13:15.805347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.805285    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" containerName="main"
May 25 12:13:15.805347 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.805299    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="0ada20ec-3961-4c62-9c41-55d7a22a519d" containerName="main"
May 25 12:13:15.808166 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.808146    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlrfx/must-gather-cq7bg"
May 25 12:13:15.811989 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.811966    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mlrfx\"/\"kube-root-ca.crt\""
May 25 12:13:15.812235 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.812216    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mlrfx\"/\"default-dockercfg-wwzqc\""
May 25 12:13:15.812310 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.812226    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mlrfx\"/\"openshift-service-ca.crt\""
May 25 12:13:15.819191 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.819168    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlrfx/must-gather-cq7bg"]
May 25 12:13:15.902169 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.902120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbpf\" (UniqueName: \"kubernetes.io/projected/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-kube-api-access-gkbpf\") pod \"must-gather-cq7bg\" (UID: \"29f9f1d5-8afa-41e0-b9a4-39cc7db80881\") " pod="openshift-must-gather-mlrfx/must-gather-cq7bg"
May 25 12:13:15.902169 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:15.902171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-must-gather-output\") pod \"must-gather-cq7bg\" (UID: \"29f9f1d5-8afa-41e0-b9a4-39cc7db80881\") " pod="openshift-must-gather-mlrfx/must-gather-cq7bg"
May 25 12:13:16.002951 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:16.002915    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbpf\" (UniqueName: \"kubernetes.io/projected/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-kube-api-access-gkbpf\") pod \"must-gather-cq7bg\" (UID: \"29f9f1d5-8afa-41e0-b9a4-39cc7db80881\") " pod="openshift-must-gather-mlrfx/must-gather-cq7bg"
May 25 12:13:16.003133 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:16.002961    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-must-gather-output\") pod \"must-gather-cq7bg\" (UID: \"29f9f1d5-8afa-41e0-b9a4-39cc7db80881\") " pod="openshift-must-gather-mlrfx/must-gather-cq7bg"
May 25 12:13:16.003353 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:16.003328    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-must-gather-output\") pod \"must-gather-cq7bg\" (UID: \"29f9f1d5-8afa-41e0-b9a4-39cc7db80881\") " pod="openshift-must-gather-mlrfx/must-gather-cq7bg"
May 25 12:13:16.012163 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:16.012139    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbpf\" (UniqueName: \"kubernetes.io/projected/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-kube-api-access-gkbpf\") pod \"must-gather-cq7bg\" (UID: \"29f9f1d5-8afa-41e0-b9a4-39cc7db80881\") " pod="openshift-must-gather-mlrfx/must-gather-cq7bg"
May 25 12:13:16.117805 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:16.117726    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlrfx/must-gather-cq7bg"
May 25 12:13:16.248685 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:16.248658    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlrfx/must-gather-cq7bg"]
May 25 12:13:16.250394 ip-10-0-133-58 kubenswrapper[2578]: W0525 12:13:16.250365    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29f9f1d5_8afa_41e0_b9a4_39cc7db80881.slice/crio-87879870cf7a33db49abd6d1ab398c6aa98338a0415f868dbea6fcbb310340d5 WatchSource:0}: Error finding container 87879870cf7a33db49abd6d1ab398c6aa98338a0415f868dbea6fcbb310340d5: Status 404 returned error can't find the container with id 87879870cf7a33db49abd6d1ab398c6aa98338a0415f868dbea6fcbb310340d5
May 25 12:13:16.378481 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:16.378368    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlrfx/must-gather-cq7bg" event={"ID":"29f9f1d5-8afa-41e0-b9a4-39cc7db80881","Type":"ContainerStarted","Data":"87879870cf7a33db49abd6d1ab398c6aa98338a0415f868dbea6fcbb310340d5"}
May 25 12:13:16.378640 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:16.378488    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 25 12:13:16.401428 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:16.401392    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 12:13:16.405349 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:16.405315    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 25 12:13:16.937180 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:16.937138    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27a8975d-9fa8-4687-95d2-7b4bcbeaed17" path="/var/lib/kubelet/pods/27a8975d-9fa8-4687-95d2-7b4bcbeaed17/volumes"
May 25 12:13:22.406034 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:22.405971    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlrfx/must-gather-cq7bg" event={"ID":"29f9f1d5-8afa-41e0-b9a4-39cc7db80881","Type":"ContainerStarted","Data":"45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec"}
May 25 12:13:23.411750 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:23.411704    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlrfx/must-gather-cq7bg" event={"ID":"29f9f1d5-8afa-41e0-b9a4-39cc7db80881","Type":"ContainerStarted","Data":"a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d"}
May 25 12:13:23.429941 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:23.429884    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-mlrfx/must-gather-cq7bg" podStartSLOduration=2.477022954 podStartE2EDuration="8.429867955s" podCreationTimestamp="2026-05-25 12:13:15 +0000 UTC" firstStartedPulling="2026-05-25 12:13:16.25216954 +0000 UTC m=+3055.901454526" lastFinishedPulling="2026-05-25 12:13:22.205014551 +0000 UTC m=+3061.854299527" observedRunningTime="2026-05-25 12:13:23.427524371 +0000 UTC m=+3063.076809367" watchObservedRunningTime="2026-05-25 12:13:23.429867955 +0000 UTC m=+3063.079152977"
May 25 12:13:27.310887 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:27.310851    2578 scope.go:117] "RemoveContainer" containerID="66b1976700d1d6971275e0f7dd96dbd99dd34bf76e61d2518adf4d8367412a8b"
May 25 12:13:31.677090 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:31.677056    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:32.621475 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:32.621423    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:33.579788 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:33.579755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:34.507471 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:34.507431    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:35.456742 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:35.456711    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:36.369702 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:36.369673    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:37.295509 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:37.295479    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:38.207649 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:38.207619    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:39.139754 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:39.139723    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:40.064502 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:40.064475    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:40.975214 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:40.975188    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:41.941646 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:41.941620    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:42.929503 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:42.929475    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:43.944157 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:43.944124    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-7bb57db77b-bxmcp_654c9dd8-9fec-4981-8a90-59ae16087eb2/istio-proxy/0.log"
May 25 12:13:46.645612 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:46.645577    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-mgvhp_39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d/authorino/0.log"
May 25 12:13:46.672120 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:46.672091    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-nkqx5_909938be-ed0b-4f6d-9819-b8ba2391790f/manager/0.log"
May 25 12:13:46.686441 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:46.686415    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-8bs8n_cb3d9513-2578-471c-a5c5-b18f4a40261d/kuadrant-console-plugin/0.log"
May 25 12:13:47.509647 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:47.509609    2578 generic.go:358] "Generic (PLEG): container finished" podID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" containerID="45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec" exitCode=0
May 25 12:13:47.509894 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:47.509687    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlrfx/must-gather-cq7bg" event={"ID":"29f9f1d5-8afa-41e0-b9a4-39cc7db80881","Type":"ContainerDied","Data":"45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec"}
May 25 12:13:47.510109 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:47.510092    2578 scope.go:117] "RemoveContainer" containerID="45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec"
May 25 12:13:48.442976 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:48.442951    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mlrfx_must-gather-cq7bg_29f9f1d5-8afa-41e0-b9a4-39cc7db80881/gather/0.log"
May 25 12:13:51.703007 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:51.702974    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vcnp5_9d037290-6471-4e48-bf2e-bcaa7de9cc64/global-pull-secret-syncer/0.log"
May 25 12:13:51.740228 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:51.740192    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6xxd2_7080cc52-e076-45dc-8653-d4e259dc912e/konnectivity-agent/0.log"
May 25 12:13:51.859350 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:51.859321    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-58.ec2.internal_ea7fcd2d26b56e02b20df01f9c251236/haproxy/0.log"
May 25 12:13:53.991263 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:53.991228    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-mlrfx/must-gather-cq7bg"]
May 25 12:13:53.991848 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:53.991534    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-mlrfx/must-gather-cq7bg" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" containerName="copy" containerID="cri-o://a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d" gracePeriod=2
May 25 12:13:53.993358 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:53.993329    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-mlrfx/must-gather-cq7bg"]
May 25 12:13:53.994515 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:53.994482    2578 status_manager.go:895] "Failed to get status for pod" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" pod="openshift-must-gather-mlrfx/must-gather-cq7bg" err="pods \"must-gather-cq7bg\" is forbidden: User \"system:node:ip-10-0-133-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mlrfx\": no relationship found between node 'ip-10-0-133-58.ec2.internal' and this object"
May 25 12:13:54.226071 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.226046    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mlrfx_must-gather-cq7bg_29f9f1d5-8afa-41e0-b9a4-39cc7db80881/copy/0.log"
May 25 12:13:54.226437 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.226416    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlrfx/must-gather-cq7bg"
May 25 12:13:54.229106 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.229081    2578 status_manager.go:895] "Failed to get status for pod" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" pod="openshift-must-gather-mlrfx/must-gather-cq7bg" err="pods \"must-gather-cq7bg\" is forbidden: User \"system:node:ip-10-0-133-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mlrfx\": no relationship found between node 'ip-10-0-133-58.ec2.internal' and this object"
May 25 12:13:54.231388 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.231367    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-must-gather-output\") pod \"29f9f1d5-8afa-41e0-b9a4-39cc7db80881\" (UID: \"29f9f1d5-8afa-41e0-b9a4-39cc7db80881\") "
May 25 12:13:54.231522 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.231398    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkbpf\" (UniqueName: \"kubernetes.io/projected/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-kube-api-access-gkbpf\") pod \"29f9f1d5-8afa-41e0-b9a4-39cc7db80881\" (UID: \"29f9f1d5-8afa-41e0-b9a4-39cc7db80881\") "
May 25 12:13:54.233852 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.233827    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-kube-api-access-gkbpf" (OuterVolumeSpecName: "kube-api-access-gkbpf") pod "29f9f1d5-8afa-41e0-b9a4-39cc7db80881" (UID: "29f9f1d5-8afa-41e0-b9a4-39cc7db80881"). InnerVolumeSpecName "kube-api-access-gkbpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:13:54.236741 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.236714    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "29f9f1d5-8afa-41e0-b9a4-39cc7db80881" (UID: "29f9f1d5-8afa-41e0-b9a4-39cc7db80881"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:13:54.332925 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.332825    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-must-gather-output\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:13:54.332925 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.332867    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkbpf\" (UniqueName: \"kubernetes.io/projected/29f9f1d5-8afa-41e0-b9a4-39cc7db80881-kube-api-access-gkbpf\") on node \"ip-10-0-133-58.ec2.internal\" DevicePath \"\""
May 25 12:13:54.536285 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.536254    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mlrfx_must-gather-cq7bg_29f9f1d5-8afa-41e0-b9a4-39cc7db80881/copy/0.log"
May 25 12:13:54.536650 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.536627    2578 generic.go:358] "Generic (PLEG): container finished" podID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" containerID="a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d" exitCode=143
May 25 12:13:54.536721 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.536685    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlrfx/must-gather-cq7bg"
May 25 12:13:54.536778 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.536741    2578 scope.go:117] "RemoveContainer" containerID="a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d"
May 25 12:13:54.541545 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.541510    2578 status_manager.go:895] "Failed to get status for pod" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" pod="openshift-must-gather-mlrfx/must-gather-cq7bg" err="pods \"must-gather-cq7bg\" is forbidden: User \"system:node:ip-10-0-133-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mlrfx\": no relationship found between node 'ip-10-0-133-58.ec2.internal' and this object"
May 25 12:13:54.545231 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.545214    2578 scope.go:117] "RemoveContainer" containerID="45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec"
May 25 12:13:54.549155 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.549130    2578 status_manager.go:895] "Failed to get status for pod" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" pod="openshift-must-gather-mlrfx/must-gather-cq7bg" err="pods \"must-gather-cq7bg\" is forbidden: User \"system:node:ip-10-0-133-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mlrfx\": no relationship found between node 'ip-10-0-133-58.ec2.internal' and this object"
May 25 12:13:54.560870 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.560851    2578 scope.go:117] "RemoveContainer" containerID="a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d"
May 25 12:13:54.561167 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:13:54.561143    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d\": container with ID starting with a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d not found: ID does not exist" containerID="a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d"
May 25 12:13:54.561271 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.561176    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d"} err="failed to get container status \"a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d\": rpc error: code = NotFound desc = could not find container \"a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d\": container with ID starting with a62978380ac2a715cabdcda9d49e4e5e3885376dda15881e08ca650a13a7f54d not found: ID does not exist"
May 25 12:13:54.561271 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.561197    2578 scope.go:117] "RemoveContainer" containerID="45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec"
May 25 12:13:54.561514 ip-10-0-133-58 kubenswrapper[2578]: E0525 12:13:54.561490    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec\": container with ID starting with 45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec not found: ID does not exist" containerID="45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec"
May 25 12:13:54.561609 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.561517    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec"} err="failed to get container status \"45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec\": rpc error: code = NotFound desc = could not find container \"45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec\": container with ID starting with 45ea63150895ff5d14c1a914be36b9865c9bbec2645aab23e763cdda9adaceec not found: ID does not exist"
May 25 12:13:54.935576 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:54.935543    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" path="/var/lib/kubelet/pods/29f9f1d5-8afa-41e0-b9a4-39cc7db80881/volumes"
May 25 12:13:55.711578 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:55.711545    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-mgvhp_39cfa8b7-72d6-44fe-8a85-5f4e05d7d26d/authorino/0.log"
May 25 12:13:55.792716 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:55.792679    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-nkqx5_909938be-ed0b-4f6d-9819-b8ba2391790f/manager/0.log"
May 25 12:13:55.829192 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:55.829164    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-8bs8n_cb3d9513-2578-471c-a5c5-b18f4a40261d/kuadrant-console-plugin/0.log"
May 25 12:13:57.321950 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:57.321924    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bh2c7_dd04e54a-1fc4-4a20-a894-76b3935e0bdb/node-exporter/0.log"
May 25 12:13:57.349181 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:57.349156    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bh2c7_dd04e54a-1fc4-4a20-a894-76b3935e0bdb/kube-rbac-proxy/0.log"
May 25 12:13:57.374091 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:13:57.374067    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bh2c7_dd04e54a-1fc4-4a20-a894-76b3935e0bdb/init-textfile/0.log"
May 25 12:14:00.602484 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.602436    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"]
May 25 12:14:00.602870 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.602790    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" containerName="gather"
May 25 12:14:00.602870 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.602804    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" containerName="gather"
May 25 12:14:00.602870 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.602822    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" containerName="copy"
May 25 12:14:00.602870 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.602829    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" containerName="copy"
May 25 12:14:00.603004 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.602881    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" containerName="copy"
May 25 12:14:00.603004 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.602897    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="29f9f1d5-8afa-41e0-b9a4-39cc7db80881" containerName="gather"
May 25 12:14:00.604688 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.604671    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.611747 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.611724    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w6vp7\"/\"openshift-service-ca.crt\""
May 25 12:14:00.612644 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.612628    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w6vp7\"/\"kube-root-ca.crt\""
May 25 12:14:00.612710 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.612669    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w6vp7\"/\"default-dockercfg-fw7vm\""
May 25 12:14:00.634547 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.634516    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"]
May 25 12:14:00.681025 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.680982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-sys\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.681197 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.681038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98s7\" (UniqueName: \"kubernetes.io/projected/3a68c79d-5677-437e-9df7-82869cba60fd-kube-api-access-b98s7\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.681197 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.681113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-proc\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.681197 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.681142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-lib-modules\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.681197 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.681178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-podres\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.781846 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.781803    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b98s7\" (UniqueName: \"kubernetes.io/projected/3a68c79d-5677-437e-9df7-82869cba60fd-kube-api-access-b98s7\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.782026 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.781869    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-proc\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.782026 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.781897    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-lib-modules\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.782026 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.781950    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-podres\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.782026 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.781995    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-sys\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.782026 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.782006    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-proc\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.782193 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.782079    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-sys\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.782193 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.782103    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-podres\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.782193 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.782108    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a68c79d-5677-437e-9df7-82869cba60fd-lib-modules\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.791711 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.791686    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98s7\" (UniqueName: \"kubernetes.io/projected/3a68c79d-5677-437e-9df7-82869cba60fd-kube-api-access-b98s7\") pod \"perf-node-gather-daemonset-lkm2j\" (UID: \"3a68c79d-5677-437e-9df7-82869cba60fd\") " pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:00.915076 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:00.914968    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:01.047086 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:01.047003    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"]
May 25 12:14:01.049606 ip-10-0-133-58 kubenswrapper[2578]: W0525 12:14:01.049572    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3a68c79d_5677_437e_9df7_82869cba60fd.slice/crio-e88c41dea8c8e676d7e348ebd293e708646e4765eb59cb2810dc986c17b7233d WatchSource:0}: Error finding container e88c41dea8c8e676d7e348ebd293e708646e4765eb59cb2810dc986c17b7233d: Status 404 returned error can't find the container with id e88c41dea8c8e676d7e348ebd293e708646e4765eb59cb2810dc986c17b7233d
May 25 12:14:01.564343 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:01.564300    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j" event={"ID":"3a68c79d-5677-437e-9df7-82869cba60fd","Type":"ContainerStarted","Data":"ec71107cde6098a41f285ff9bc44440d718745af35edc45591a87f1f767d6ed4"}
May 25 12:14:01.564343 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:01.564346    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j" event={"ID":"3a68c79d-5677-437e-9df7-82869cba60fd","Type":"ContainerStarted","Data":"e88c41dea8c8e676d7e348ebd293e708646e4765eb59cb2810dc986c17b7233d"}
May 25 12:14:01.564638 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:01.564436    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:01.585621 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:01.585574    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j" podStartSLOduration=1.585558191 podStartE2EDuration="1.585558191s" podCreationTimestamp="2026-05-25 12:14:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:14:01.582065992 +0000 UTC m=+3101.231350996" watchObservedRunningTime="2026-05-25 12:14:01.585558191 +0000 UTC m=+3101.234843185"
May 25 12:14:01.733143 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:01.733113    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rlr4p_f0720f3e-ce4d-4fc2-a5de-5b624520ae3c/dns/0.log"
May 25 12:14:01.756165 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:01.756139    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rlr4p_f0720f3e-ce4d-4fc2-a5de-5b624520ae3c/kube-rbac-proxy/0.log"
May 25 12:14:01.836746 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:01.836668    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wkkdr_6375e270-ce2d-4549-a415-343be34c4dfc/dns-node-resolver/0.log"
May 25 12:14:02.397645 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:02.397617    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-57jwf_ac7c52f6-4dec-48a3-a106-0fb515461899/node-ca/0.log"
May 25 12:14:03.856647 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:03.856619    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h5vl5_f6151b3e-2de0-49db-a40a-3a4d0e75c3e1/serve-healthcheck-canary/0.log"
May 25 12:14:04.531324 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:04.531291    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xl9q7_6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf/kube-rbac-proxy/0.log"
May 25 12:14:04.552444 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:04.552417    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xl9q7_6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf/exporter/0.log"
May 25 12:14:04.576778 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:04.576745    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xl9q7_6a306024-ef9e-4f8d-bdd5-4a77fbf6eeaf/extractor/0.log"
May 25 12:14:07.150111 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:07.150088    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8646484d7f-glghm_63aa5fd3-ccd9-497f-90c8-e023f06b7dfe/manager/0.log"
May 25 12:14:07.581639 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:07.581612    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w6vp7/perf-node-gather-daemonset-lkm2j"
May 25 12:14:07.827758 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:07.827732    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-2sqrq_96002196-66a8-4257-aa19-6ac0b2bf6a05/server/0.log"
May 25 12:14:08.024249 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:08.024201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-8xqts_9a3e73f0-be2e-4c2d-9688-0ef36119fe6d/manager/0.log"
May 25 12:14:08.044317 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:08.044291    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-h56gh_241b75d1-a355-4b2b-84d2-4bf4e81ad0ac/s3-init/0.log"
May 25 12:14:08.072842 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:08.072811    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-dbwql_12dfe165-4edb-48a2-8b52-34d653fa7989/seaweedfs/0.log"
May 25 12:14:14.337156 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:14.337114    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9jz7_62c640d4-74b5-4e3c-b772-f4e818c25bfd/kube-multus-additional-cni-plugins/0.log"
May 25 12:14:14.360185 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:14.360154    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9jz7_62c640d4-74b5-4e3c-b772-f4e818c25bfd/egress-router-binary-copy/0.log"
May 25 12:14:14.383351 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:14.383319    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9jz7_62c640d4-74b5-4e3c-b772-f4e818c25bfd/cni-plugins/0.log"
May 25 12:14:14.405853 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:14.405829    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9jz7_62c640d4-74b5-4e3c-b772-f4e818c25bfd/bond-cni-plugin/0.log"
May 25 12:14:14.426931 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:14.426909    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9jz7_62c640d4-74b5-4e3c-b772-f4e818c25bfd/routeoverride-cni/0.log"
May 25 12:14:14.448291 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:14.448264    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9jz7_62c640d4-74b5-4e3c-b772-f4e818c25bfd/whereabouts-cni-bincopy/0.log"
May 25 12:14:14.473102 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:14.473075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l9jz7_62c640d4-74b5-4e3c-b772-f4e818c25bfd/whereabouts-cni/0.log"
May 25 12:14:14.533593 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:14.533569    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wdc9d_e93f1f05-2be2-471a-9f26-73d201d99aca/kube-multus/0.log"
May 25 12:14:14.637108 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:14.637032    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tpl95_489b5117-49e0-45e5-b99a-3ad57febc838/network-metrics-daemon/0.log"
May 25 12:14:14.657662 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:14.657634    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tpl95_489b5117-49e0-45e5-b99a-3ad57febc838/kube-rbac-proxy/0.log"
May 25 12:14:16.072795 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:16.072766    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-controller/0.log"
May 25 12:14:16.092174 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:16.092143    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/0.log"
May 25 12:14:16.105652 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:16.105542    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovn-acl-logging/1.log"
May 25 12:14:16.125825 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:16.125789    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/kube-rbac-proxy-node/0.log"
May 25 12:14:16.148080 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:16.148051    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/kube-rbac-proxy-ovn-metrics/0.log"
May 25 12:14:16.171829 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:16.171803    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/northd/0.log"
May 25 12:14:16.201804 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:16.201779    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/nbdb/0.log"
May 25 12:14:16.251181 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:16.251147    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/sbdb/0.log"
May 25 12:14:16.378185 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:16.378113    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pvk9r_1575503e-b475-4021-822e-f2e700cfbf03/ovnkube-controller/0.log"
May 25 12:14:17.501220 ip-10-0-133-58 kubenswrapper[2578]: I0525 12:14:17.501172    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9xqkb_d0e477d9-b58a-4766-b341-ef28fa5d69ab/network-check-target-container/0.log"