Jul 07 02:17:44.482634 ip-10-0-141-229 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 02:17:44.860667 ip-10-0-141-229 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:17:44.860667 ip-10-0-141-229 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 02:17:44.860667 ip-10-0-141-229 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:17:44.860667 ip-10-0-141-229 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 02:17:44.860667 ip-10-0-141-229 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:17:44.861316 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.860745    2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 02:17:44.866188 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866172    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:44.866188 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866188    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866192    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866195    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866198    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866202    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866204    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866207    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866210    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866212    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866217    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866221    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866225    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866228    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866231    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866233    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866236    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866240    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866243    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866246    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:44.866263 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866248    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866251    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866254    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866256    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866259    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866262    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866264    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866267    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866269    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866272    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866275    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866278    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866280    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866282    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866285    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866287    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866290    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866292    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866295    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866297    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:44.866740 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866299    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866302    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866304    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866307    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866312    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866315    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866318    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866320    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866322    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866325    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866327    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866329    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866332    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866334    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866337    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866339    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866359    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866362    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866366    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:44.867218 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866368    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866372    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866374    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866377    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866381    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866384    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866387    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866390    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866392    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866395    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866397    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866400    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866402    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866405    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866407    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866409    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866412    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866414    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866418    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866421    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:44.867698 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866423    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866426    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866428    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866431    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866433    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866436    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866439    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866443    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866446    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866448    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866451    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866453    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866456    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866459    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866461    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866464    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866466    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866469    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866471    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.866473    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:44.868168 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867165    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867171    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867175    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867177    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867180    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867182    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867185    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867188    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867190    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867192    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867195    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867198    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867200    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867203    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867205    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867208    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867210    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867215    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867219    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:44.868767 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867223    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867226    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867229    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867231    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867234    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867237    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867239    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867242    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867244    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867246    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867249    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867252    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867254    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867256    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867259    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867261    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867263    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867266    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867268    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:44.869253 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867272    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867276    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867279    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867282    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867284    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867287    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867290    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867292    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867294    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867297    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867299    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867302    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867304    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867307    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867310    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867312    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867315    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867318    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867320    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867322    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:44.869748 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867325    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867327    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867329    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867332    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867336    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867339    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867363    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867368    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867371    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867373    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867376    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867379    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867382    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867385    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867388    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867391    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867394    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867396    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867399    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867402    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:44.870235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867406    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867408    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867411    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867413    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867417    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867420    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867422    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867425    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867427    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867430    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867432    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867434    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867437    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867439    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867442    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867444    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867446    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867449    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867451    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867454    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:44.870756 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867456    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867820    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867825    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867828    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867830    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867833    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867835    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867838    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867840    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867843    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867845    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867848    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867851    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867854    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867856    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867859    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867862    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867865    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867868    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867871    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:44.871238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867873    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867875    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867878    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867880    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867883    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867885    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867888    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867891    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867893    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867896    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867898    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867900    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867903    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867905    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867907    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867910    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867912    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867914    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867918    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867921    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:44.871770 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867924    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867926    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867929    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867931    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867934    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867936    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867939    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867941    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867944    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867948    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867950    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867953    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867955    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867957    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867960    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867962    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867965    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867967    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867970    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:44.872261 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867972    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867975    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867977    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867980    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867982    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867984    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867987    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867989    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867991    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867994    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.867996    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868000    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868004    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868007    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868010    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868012    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868015    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868019    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868023    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:44.872737 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868025    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868028    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868032    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868034    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868037    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868040    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868042    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868044    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868047    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868049    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868052    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868054    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868056    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868059    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868061    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868064    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868066    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868068    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868071    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868074    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:44.873237 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868076    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868079    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868151    2570 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868158    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868163    2570 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868168    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868172    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868175    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868179    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868183    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868186    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868189    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868193    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868196    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868200    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868202    2570 flags.go:64] FLAG: --cgroup-root=""
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868205    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868208    2570 flags.go:64] FLAG: --client-ca-file=""
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868211    2570 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868213    2570 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868218    2570 flags.go:64] FLAG: --cluster-domain=""
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868220    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868223    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868226    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 02:17:44.873755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868229    2570 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868233    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868235    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868238    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868241    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868244    2570 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868246    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868251    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868254    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868257    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868260    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868263    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868266    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868268    2570 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868271    2570 flags.go:64] FLAG: --enable-server="true"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868274    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868277    2570 flags.go:64] FLAG: --event-burst="100"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868280    2570 flags.go:64] FLAG: --event-qps="50"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868283    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868285    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868288    2570 flags.go:64] FLAG: --eviction-hard=""
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868293    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868295    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868298    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868301    2570 flags.go:64] FLAG: --eviction-soft=""
Jul 07 02:17:44.874409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868304    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868307    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868309    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868312    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868315    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868317    2570 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868320    2570 flags.go:64] FLAG: --feature-gates=""
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868324    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868327    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868330    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868333    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868335    2570 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868338    2570 flags.go:64] FLAG: --help="false"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868341    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-141-229.ec2.internal"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868363    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868366    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868369    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868372    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868375    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868378    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868381    2570 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868383    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868386    2570 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 02:17:44.875030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868389    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868392    2570 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868395    2570 flags.go:64] FLAG: --kube-reserved=""
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868398    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868401    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868404    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868407    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868410    2570 flags.go:64] FLAG: --lock-file=""
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868412    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868415    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868418    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868423    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868426    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868429    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868431    2570 flags.go:64] FLAG: --logging-format="text"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868434    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868437    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868440    2570 flags.go:64] FLAG: --manifest-url=""
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868442    2570 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868446    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868449    2570 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868453    2570 flags.go:64] FLAG: --max-pods="110"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868456    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868460    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868463    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 02:17:44.875629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868465    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868468    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868471    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868474    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868480    2570 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868483    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868486    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868489    2570 flags.go:64] FLAG: --pod-cidr=""
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868492    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868497    2570 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868500    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868502    2570 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868505    2570 flags.go:64] FLAG: --port="10250"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868508    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868511    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03e3d34289ac21074"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868514    2570 flags.go:64] FLAG: --qos-reserved=""
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868517    2570 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868520    2570 flags.go:64] FLAG: --register-node="true"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868522    2570 flags.go:64] FLAG: --register-with-taints=""
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868526    2570 flags.go:64] FLAG: --registry-burst="10"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868528    2570 flags.go:64] FLAG: --registry-qps="5"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868531    2570 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868533    2570 flags.go:64] FLAG: --reserved-memory=""
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868537    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868539    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 02:17:44.876239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868542    2570 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868545    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868547    2570 flags.go:64] FLAG: --runonce="false"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868550    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868553    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868557    2570 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868559    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868562    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868565    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868568    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868571    2570 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868574    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868577    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868579    2570 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868582    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868585    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868589    2570 flags.go:64] FLAG: --system-cgroups=""
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868592    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868597    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868599    2570 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868602    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868606    2570 flags.go:64] FLAG: --tls-min-version=""
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868609    2570 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868612    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868615    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 02:17:44.876872 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868618    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868620    2570 flags.go:64] FLAG: --v="2"
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868624    2570 flags.go:64] FLAG: --version="false"
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868628    2570 flags.go:64] FLAG: --vmodule=""
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868631    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.868634    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868743    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868746    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868749    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868752    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868754    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868757    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868761    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868765    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868769    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868771    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868774    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868777    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868779    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868783    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:44.877512 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868787    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868790    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868792    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868797    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868800    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868803    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868805    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868808    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868811    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868813    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868816    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868827    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868830    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868833    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868835    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868838    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868840    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868842    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868845    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868847    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:44.878014 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868850    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868852    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868855    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868857    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868861    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868863    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868866    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868868    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868871    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868873    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868876    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868878    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868880    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868883    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868885    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868889    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868891    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868894    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868896    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868899    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:44.878611 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868901    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868903    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868906    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868909    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868911    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868914    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868916    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868919    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868921    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868924    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868926    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868928    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868931    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868933    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868935    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868938    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868941    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868944    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868946    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868949    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:44.879112 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868951    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868954    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868956    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868959    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868961    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868963    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868966    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868970    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868972    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868974    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868977    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868979    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868982    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868984    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868987    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868989    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868994    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.868997    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.869000    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:44.879629 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.869002    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.869005    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.869007    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.869009    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.869012    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.869014    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.869541    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.876074    2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.876089    2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.876105    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.876111    2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876170    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876175    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876179    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876182    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:44.880091 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876185    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876187    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876190    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876193    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876195    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876198    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876200    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876203    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876205    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876208    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876210    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876213    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876215    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876217    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876226    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876228    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876231    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876233    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876236    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876238    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:44.880476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876241    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876244    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876246    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876249    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876251    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876255    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876258    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876262    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876266    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876269    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876272    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876275    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876278    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876280    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876283    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876286    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876288    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876291    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876293    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876296    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:44.880959 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876299    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876301    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876304    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876306    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876309    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876311    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876313    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876321    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876324    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876326    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876329    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876331    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876333    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876340    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876360    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876364    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876366    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876369    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876372    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:44.881478 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876375    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876377    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876380    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876382    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876385    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876387    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876390    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876392    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876394    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876397    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876399    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876402    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876404    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876406    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876409    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876412    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876415    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876419    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876421    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:44.881988 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876424    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876427    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876436    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876439    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876441    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876444    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876447    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876449    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876452    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876454    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876456    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876460    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876463    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876465    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876467    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876475    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:44.882479 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876478    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.876483    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876621    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876626    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876628    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876632    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876635    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876638    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876641    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876643    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876646    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876648    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876651    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876654    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:17:44.882873 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876656    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876658    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876661    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876663    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876666    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876674    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876678    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876681    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876683    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876686    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876689    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876691    2570 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876694    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876697    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876699    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876702    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876704    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876707    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876710    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:17:44.883235 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876712    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876715    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876717    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876719    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876722    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876724    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876726    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876729    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876731    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876733    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876736    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876738    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876740    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876743    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876745    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876748    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876750    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876752    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876755    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876763    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:17:44.883820 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876765    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876768    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876770    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876772    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876775    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876777    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876780    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876783    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876785    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876787    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876790    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876792    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876795    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876797    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876799    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876802    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876804    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876806    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876809    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876811    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:17:44.884647 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876813    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876816    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876826    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876829    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876831    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876834    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876836    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876839    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876841    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876844    2570 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876846    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876848    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876857    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876860    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876862    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876864    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876867    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876869    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876872    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876874    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:17:44.885214 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876877    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876880    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876882    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876885    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876887    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876890    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876892    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:44.876895    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.876900    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.876998    2570 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.878952    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.880370    2570 server.go:1019] "Starting client certificate rotation"
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.880465    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.880500    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 02:17:44.885987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.882875    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 02:17:44.886337 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.884365    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 02:17:44.898374 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.898355    2570 log.go:25] "Validated CRI v1 runtime API"
Jul 07 02:17:44.900748 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.900729    2570 log.go:25] "Validated CRI v1 image API"
Jul 07 02:17:44.902056 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.902041    2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 02:17:44.904260 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.904240    2570 fs.go:135] Filesystem UUIDs: map[0d2d9cff-053f-46d3-b0ce-9bd7794a8869:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8c81e7f0-d177-4d8c-90ee-d5f333a9b858:/dev/nvme0n1p4]
Jul 07 02:17:44.904362 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.904261    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 02:17:44.909292 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.909180    2570 manager.go:216] Machine: {Timestamp:2026-07-07 02:17:44.907814436 +0000 UTC m=+0.315404114 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100385 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24d4d59b91de41294b6db9b16c7955 SystemUUID:ec24d4d5-9b91-de41-294b-6db9b16c7955 BootID:1fa21bb1-4cd2-4d2a-8754-382f59548851 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:79:05:b7:41:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:79:05:b7:41:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:f1:ac:68:3a:a2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 02:17:44.909292 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.909283    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 02:17:44.909471 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.909393    2570 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 02:17:44.909992 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.909975    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 02:17:44.911949 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.911921    2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 02:17:44.912090 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.911949    2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-229.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 02:17:44.912681 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.912671    2570 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 02:17:44.912714 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.912682    2570 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 02:17:44.912714 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.912690    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 02:17:44.913239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.913229    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 02:17:44.914492 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.914482    2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 02:17:44.917110 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.917099    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 02:17:44.917208 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.917199    2570 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 02:17:44.918034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.918022    2570 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 02:17:44.918080 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.918037    2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 02:17:44.918080 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.918053    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 02:17:44.918080 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.918062    2570 kubelet.go:398] "Adding apiserver pod source"
Jul 07 02:17:44.918080 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.918070    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 02:17:44.918396 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.918385    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 02:17:44.918430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.918403    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 02:17:44.920566 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.920550    2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 02:17:44.922217 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.922205    2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 02:17:44.922266 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.922234    2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 02:17:44.923741 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923730    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 02:17:44.923808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923744    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 02:17:44.923808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923750    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 02:17:44.923808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923755    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 02:17:44.923808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923763    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 02:17:44.923808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923769    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 02:17:44.923808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923776    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 02:17:44.923808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923781    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 02:17:44.923808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923787    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 02:17:44.923808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923792    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 02:17:44.923808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923800    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 02:17:44.923808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.923804    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 02:17:44.925136 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.925125    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 02:17:44.925136 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.925136    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 02:17:44.927937 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.927924    2570 server.go:1268] "Started kubelet"
Jul 07 02:17:44.928094 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.928036    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 02:17:44.928193 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.928034    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 02:17:44.928680 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.928663    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 02:17:44.928711 ip-10-0-141-229 systemd[1]: Started Kubernetes Kubelet.
Jul 07 02:17:44.931447 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.931415    2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 02:17:44.932580 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.932562    2570 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 02:17:44.933639 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.933612    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 02:17:44.933829 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.933774    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-229.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 02:17:44.933986 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.933963    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 02:17:44.934171 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.934135    2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 02:17:44.938735 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.938718    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 02:17:44.938811 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.938790    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 02:17:44.939004 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.938987    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:44.939437 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.939419    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 02:17:44.939437 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.939423    2570 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 02:17:44.939566 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.939447    2570 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 02:17:44.939566 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.939557    2570 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 02:17:44.939566 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.939565    2570 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 02:17:44.940199 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.940181    2570 factory.go:153] Registering CRI-O factory
Jul 07 02:17:44.940199 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.940198    2570 factory.go:223] Registration of the crio container factory successfully
Jul 07 02:17:44.940335 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.940239    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 02:17:44.940335 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.940246    2570 factory.go:55] Registering systemd factory
Jul 07 02:17:44.940335 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.940251    2570 factory.go:223] Registration of the systemd container factory successfully
Jul 07 02:17:44.940335 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.940267    2570 factory.go:103] Registering Raw factory
Jul 07 02:17:44.940335 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.940276    2570 manager.go:1196] Started watching for new ooms in manager
Jul 07 02:17:44.940726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.940713    2570 manager.go:349] Starting recovery of all containers
Jul 07 02:17:44.940880 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.939902    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-229.ec2.internal.18bfe05c66c6619d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-229.ec2.internal,UID:ip-10-0-141-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-229.ec2.internal,},FirstTimestamp:2026-07-07 02:17:44.927895965 +0000 UTC m=+0.335485643,LastTimestamp:2026-07-07 02:17:44.927895965 +0000 UTC m=+0.335485643,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-229.ec2.internal,}"
Jul 07 02:17:44.946125 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.946073    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 02:17:44.946242 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.946220    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 02:17:44.950847 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.950831    2570 manager.go:354] Recovery completed
Jul 07 02:17:44.954869 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.954855    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:44.956534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.956517    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:44.956609 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.956551    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:44.956609 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.956568    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:44.957056 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.957041    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 02:17:44.957056 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.957053    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 02:17:44.957163 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.957059    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:44.957163 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.957071    2570 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 02:17:44.957719 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.957703    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:44.957796 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.957730    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:44.957796 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.957742    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:44.958185 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.958175    2570 policy_none.go:49] "None policy: Start"
Jul 07 02:17:44.958218 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.958195    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 02:17:44.958218 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.958205    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 02:17:44.958913 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.958855    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-229.ec2.internal.18bfe05c687b623f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-229.ec2.internal,UID:ip-10-0-141-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-229.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-229.ec2.internal,},FirstTimestamp:2026-07-07 02:17:44.956535359 +0000 UTC m=+0.364125043,LastTimestamp:2026-07-07 02:17:44.956535359 +0000 UTC m=+0.364125043,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-229.ec2.internal,}"
Jul 07 02:17:44.959093 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.959083    2570 policy_none.go:47] "Start"
Jul 07 02:17:44.971179 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.971035    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-229.ec2.internal.18bfe05c687bc458  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-229.ec2.internal,UID:ip-10-0-141-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-229.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-229.ec2.internal,},FirstTimestamp:2026-07-07 02:17:44.956560472 +0000 UTC m=+0.364150154,LastTimestamp:2026-07-07 02:17:44.956560472 +0000 UTC m=+0.364150154,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-229.ec2.internal,}"
Jul 07 02:17:44.984841 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.984767    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-229.ec2.internal.18bfe05c687bf83e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-229.ec2.internal,UID:ip-10-0-141-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-229.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-229.ec2.internal,},FirstTimestamp:2026-07-07 02:17:44.956573758 +0000 UTC m=+0.364163438,LastTimestamp:2026-07-07 02:17:44.956573758 +0000 UTC m=+0.364163438,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-229.ec2.internal,}"
Jul 07 02:17:44.996667 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.996605    2570 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-229.ec2.internal.18bfe05c687b623f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-229.ec2.internal.18bfe05c687b623f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-229.ec2.internal,UID:ip-10-0-141-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-229.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-229.ec2.internal,},FirstTimestamp:2026-07-07 02:17:44.956535359 +0000 UTC m=+0.364125043,LastTimestamp:2026-07-07 02:17:44.957717272 +0000 UTC m=+0.365306949,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-229.ec2.internal,}"
Jul 07 02:17:44.998937 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.998914    2570 manager.go:337] "Starting Device Plugin manager"
Jul 07 02:17:44.998937 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.998918    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dhvwn"
Jul 07 02:17:45.021843 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.998952    2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 02:17:45.021843 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.998966    2570 server.go:87] "Starting device plugin registration server"
Jul 07 02:17:45.021843 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.999170    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 02:17:45.021843 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.999181    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 02:17:45.021843 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.999321    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 02:17:45.021843 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.999424    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 02:17:45.021843 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:44.999430    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 02:17:45.021843 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.999873    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 02:17:45.021843 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:44.999941    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:45.021843 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.009967    2570 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-229.ec2.internal.18bfe05c687bc458\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-229.ec2.internal.18bfe05c687bc458  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-229.ec2.internal,UID:ip-10-0-141-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-229.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-229.ec2.internal,},FirstTimestamp:2026-07-07 02:17:44.956560472 +0000 UTC m=+0.364150154,LastTimestamp:2026-07-07 02:17:44.957737291 +0000 UTC m=+0.365326968,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-229.ec2.internal,}"
Jul 07 02:17:45.021843 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.015214    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dhvwn"
Jul 07 02:17:45.079060 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.079031    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 02:17:45.080322 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.080309    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 02:17:45.080391 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.080356    2570 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 02:17:45.080391 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.080379    2570 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 02:17:45.080479 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.080408    2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 02:17:45.082823 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.082805    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:17:45.100309 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.100291    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:45.101437 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.101422    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:45.101519 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.101449    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:45.101519 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.101459    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:45.101519 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.101479    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.111624 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.111584    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.111624 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.111608    2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-229.ec2.internal\": node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:45.147598 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.147578    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:45.181309 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.181287    2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"]
Jul 07 02:17:45.181408 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.181339    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:45.182641 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.182627    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:45.182728 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.182658    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:45.182728 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.182672    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:45.184913 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.184897    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:45.185014 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.184999    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.185061 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.185026    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:45.185588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.185568    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:45.185588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.185570    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:45.185712 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.185606    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:45.185712 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.185617    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:45.185712 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.185587    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:45.185712 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.185658    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:45.187846 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.187833    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.187892 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.187857    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:17:45.188442 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.188427    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:17:45.188519 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.188453    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:17:45.188519 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.188468    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:17:45.213906 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.213881    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-229.ec2.internal\" not found" node="ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.218200 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.218187    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-229.ec2.internal\" not found" node="ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.241517 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.241498    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd0c9e68fa1554fd01092c3bef64ba0f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-229.ec2.internal\" (UID: \"dd0c9e68fa1554fd01092c3bef64ba0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.241579 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.241519    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd320ee8d7c30c12d7b97d56aa7806c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"bd320ee8d7c30c12d7b97d56aa7806c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.241579 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.241536    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd320ee8d7c30c12d7b97d56aa7806c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"bd320ee8d7c30c12d7b97d56aa7806c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.248598 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.248582    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:45.342659 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.342638    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd0c9e68fa1554fd01092c3bef64ba0f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-229.ec2.internal\" (UID: \"dd0c9e68fa1554fd01092c3bef64ba0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.342745 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.342665    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd320ee8d7c30c12d7b97d56aa7806c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"bd320ee8d7c30c12d7b97d56aa7806c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.342745 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.342681    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd320ee8d7c30c12d7b97d56aa7806c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"bd320ee8d7c30c12d7b97d56aa7806c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.342745 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.342723    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd320ee8d7c30c12d7b97d56aa7806c5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"bd320ee8d7c30c12d7b97d56aa7806c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.342848 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.342745    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd320ee8d7c30c12d7b97d56aa7806c5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"bd320ee8d7c30c12d7b97d56aa7806c5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.342848 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.342744    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd0c9e68fa1554fd01092c3bef64ba0f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-229.ec2.internal\" (UID: \"dd0c9e68fa1554fd01092c3bef64ba0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.348758 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.348743    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:45.449844 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.449785    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:45.515963 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.515945    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.521022 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.520913    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:45.550209 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.550190    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:45.650751 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.650724    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:45.751224 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.751178    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:45.851747 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.851724    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:45.879898 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.879876    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 02:17:45.880306 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.880002    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 02:17:45.952093 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:45.952074    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 07 02:17:45.965410 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:45.965389    2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:17:46.019818 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.019766    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 02:12:44 +0000 UTC" deadline="2028-01-24 22:35:54.071862005 +0000 UTC"
Jul 07 02:17:46.019818 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.019797    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13604h18m8.052073194s"
Jul 07 02:17:46.039936 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.039913    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:46.050072 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.050056    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 02:17:46.051708 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.051692    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 07 02:17:46.066911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.066892    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 02:17:46.126238 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:46.126195    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0c9e68fa1554fd01092c3bef64ba0f.slice/crio-d4a3960a06a1c7f64b24f6ea7802d31264ff37dd98a825107e94cb7564e4e57d WatchSource:0}: Error finding container d4a3960a06a1c7f64b24f6ea7802d31264ff37dd98a825107e94cb7564e4e57d: Status 404 returned error can't find the container with id d4a3960a06a1c7f64b24f6ea7802d31264ff37dd98a825107e94cb7564e4e57d
Jul 07 02:17:46.126441 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:46.126427    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd320ee8d7c30c12d7b97d56aa7806c5.slice/crio-bd0b905e3ef93cdd5bb5e4306c05c380ea1ad89d3ebef1162f80a52fccbad9c2 WatchSource:0}: Error finding container bd0b905e3ef93cdd5bb5e4306c05c380ea1ad89d3ebef1162f80a52fccbad9c2: Status 404 returned error can't find the container with id bd0b905e3ef93cdd5bb5e4306c05c380ea1ad89d3ebef1162f80a52fccbad9c2
Jul 07 02:17:46.131308 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.131293    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:17:46.187938 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.187920    2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:17:46.525280 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.525259    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:17:46.919508 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.919488    2570 apiserver.go:52] "Watching apiserver"
Jul 07 02:17:46.926755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.926732    2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 02:17:46.927126 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.927100    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-vsdng","openshift-dns/node-resolver-h4v89","openshift-image-registry/node-ca-x54gh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal","openshift-multus/multus-additional-cni-plugins-sk7b9","openshift-multus/multus-vdkb4","openshift-multus/network-metrics-daemon-wcsrt","openshift-network-diagnostics/network-check-target-7sr2n","openshift-network-operator/iptables-alerter-7n4fq","openshift-ovn-kubernetes/ovnkube-node-vcklb","kube-system/global-pull-secret-syncer-pg4n4","kube-system/konnectivity-agent-q27h5","kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"]
Jul 07 02:17:46.929502 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.929480    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7n4fq"
Jul 07 02:17:46.931587 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.931571    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h4v89"
Jul 07 02:17:46.932328 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.932200    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.932328 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.932211    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 02:17:46.932328 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.932213    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pscfz\""
Jul 07 02:17:46.932328 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.932282    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.933844 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.933830    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:46.933928 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:46.933889    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:17:46.935988 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.935969    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r6ptz\""
Jul 07 02:17:46.936073 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.936032    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.936168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.935975    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.936394 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.936368    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:46.937672 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.937657    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.938447 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.938431    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 02:17:46.938653 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.938639    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.938987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.938970    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.939064 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.939015    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 02:17:46.939116 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.938971    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 02:17:46.939116 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.939019    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vzl5g\""
Jul 07 02:17:46.939601 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.939584    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 02:17:46.939732 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.939717    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zldg6\""
Jul 07 02:17:46.939980 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.939965    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 02:17:46.940190 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.940179    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.942441 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.942426    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:46.942522 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:46.942506    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:17:46.942577 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.942550    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.942757 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.942738    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j5zdj\""
Jul 07 02:17:46.943203 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.943186    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.944526 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.944511    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x54gh"
Jul 07 02:17:46.946653 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.946635    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mqm9k\""
Jul 07 02:17:46.946772 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.946749    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:46.946856 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.946659    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 02:17:46.947242 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.947223    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.947525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.947506    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.949120 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949102    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:46.949237 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949216    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-sysctl-d\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.949298 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-systemd\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.949423 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949307    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-tuned\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.949423 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949363    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/785f14dc-4626-4437-be21-2761e326fe7d-hosts-file\") pod \"node-resolver-h4v89\" (UID: \"785f14dc-4626-4437-be21-2761e326fe7d\") " pod="openshift-dns/node-resolver-h4v89"
Jul 07 02:17:46.949423 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949391    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-cnibin\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:46.949423 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949416    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ndsr\" (UniqueName: \"kubernetes.io/projected/549d5421-b20a-4288-88c3-c3629f566982-kube-api-access-9ndsr\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.949592 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949441    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-sysctl-conf\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.949592 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949464    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-var-lib-kubelet\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.949592 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949487    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97b6993b-5c3c-400b-85fd-2044c0ae3eff-tmp\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.949592 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949514    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsznz\" (UniqueName: \"kubernetes.io/projected/0b89780e-214b-4029-8ee8-386e60ec5437-kube-api-access-wsznz\") pod \"iptables-alerter-7n4fq\" (UID: \"0b89780e-214b-4029-8ee8-386e60ec5437\") " pod="openshift-network-operator/iptables-alerter-7n4fq"
Jul 07 02:17:46.949592 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949540    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-os-release\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:46.949592 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949576    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-run-netns\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.949827 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949620    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-etc-kubernetes\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.949827 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949651    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-modprobe-d\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.949827 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:46.949657    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:17:46.949827 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949677    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-kubernetes\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.949827 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949704    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcblw\" (UniqueName: \"kubernetes.io/projected/97b6993b-5c3c-400b-85fd-2044c0ae3eff-kube-api-access-lcblw\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.949827 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949733    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b89780e-214b-4029-8ee8-386e60ec5437-iptables-alerter-script\") pod \"iptables-alerter-7n4fq\" (UID: \"0b89780e-214b-4029-8ee8-386e60ec5437\") " pod="openshift-network-operator/iptables-alerter-7n4fq"
Jul 07 02:17:46.949827 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949754    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-var-lib-cni-bin\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.949827 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949778    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-run\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.949827 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949809    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4mdp\" (UniqueName: \"kubernetes.io/projected/785f14dc-4626-4437-be21-2761e326fe7d-kube-api-access-s4mdp\") pod \"node-resolver-h4v89\" (UID: \"785f14dc-4626-4437-be21-2761e326fe7d\") " pod="openshift-dns/node-resolver-h4v89"
Jul 07 02:17:46.950166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949841    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:46.950166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949878    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b740e4f-f613-4235-8aa1-ec671ed8208a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:46.950166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949802    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 02:17:46.950166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949936    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-284ss\""
Jul 07 02:17:46.950166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950027    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.950166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.949928    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-os-release\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.950166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950127    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 02:17:46.950166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950156    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-run-k8s-cni-cncf-io\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.950541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950190    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-lib-modules\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.950541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950234    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/785f14dc-4626-4437-be21-2761e326fe7d-tmp-dir\") pod \"node-resolver-h4v89\" (UID: \"785f14dc-4626-4437-be21-2761e326fe7d\") " pod="openshift-dns/node-resolver-h4v89"
Jul 07 02:17:46.950541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950294    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-system-cni-dir\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.950541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950370    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-multus-socket-dir-parent\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.950541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950395    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-host\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.950541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950424    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-system-cni-dir\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:46.950541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950449    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b740e4f-f613-4235-8aa1-ec671ed8208a-cni-binary-copy\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:46.950541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950486    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:46.950541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950515    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg92v\" (UniqueName: \"kubernetes.io/projected/2b740e4f-f613-4235-8aa1-ec671ed8208a-kube-api-access-wg92v\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950556    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-cnibin\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950582    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950593    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950593    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-var-lib-cni-multus\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950632    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-var-lib-kubelet\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950661    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b89780e-214b-4029-8ee8-386e60ec5437-host-slash\") pod \"iptables-alerter-7n4fq\" (UID: \"0b89780e-214b-4029-8ee8-386e60ec5437\") " pod="openshift-network-operator/iptables-alerter-7n4fq"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950722    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b740e4f-f613-4235-8aa1-ec671ed8208a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950750    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-multus-cni-dir\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950758    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950771    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/549d5421-b20a-4288-88c3-c3629f566982-cni-binary-copy\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950780    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950787    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-hostroot\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950847    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-run-multus-certs\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950889    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-sysconfig\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-sys\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:46.951641 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950932    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrkd\" (UniqueName: \"kubernetes.io/projected/e49d21df-0b01-4845-ad69-b2ad83b93187-kube-api-access-rkrkd\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:46.951641 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950966    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-multus-conf-dir\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.951641 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.950983    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/549d5421-b20a-4288-88c3-c3629f566982-multus-daemon-config\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:46.951641 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.951405    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:17:46.953529 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.953512    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:46.954113 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.954094    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 02:17:46.954189 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.954141    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 02:17:46.954189 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.954182    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w25kw\""
Jul 07 02:17:46.955530 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.955513    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 02:17:46.955893 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.955879    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 02:17:46.955965 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.955880    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 02:17:46.956016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.956010    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m2c4p\""
Jul 07 02:17:46.974843 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.974827    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v6bjt"
Jul 07 02:17:46.982063 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:46.982037    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v6bjt"
Jul 07 02:17:47.015236 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.015198    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:17:47.040558 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.040540    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 02:17:47.051407 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051389    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcblw\" (UniqueName: \"kubernetes.io/projected/97b6993b-5c3c-400b-85fd-2044c0ae3eff-kube-api-access-lcblw\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.051513 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051415    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.051513 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051433    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-var-lib-cni-bin\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.051513 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051455    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68468352-f49f-497c-b807-a5e0d7e50082-env-overrides\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.051513 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051499    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051521    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-os-release\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051532    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-var-lib-cni-bin\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051539    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-run-k8s-cni-cncf-io\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051573    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-lib-modules\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051597    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-os-release\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051599    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/785f14dc-4626-4437-be21-2761e326fe7d-tmp-dir\") pod \"node-resolver-h4v89\" (UID: \"785f14dc-4626-4437-be21-2761e326fe7d\") " pod="openshift-dns/node-resolver-h4v89"
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.051619    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051630    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-run-k8s-cni-cncf-io\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051642    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdg6\" (UniqueName: \"kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6\") pod \"network-check-target-7sr2n\" (UID: \"c2b43d9f-097e-4e87-a0a6-40546d9005fe\") " pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051679    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7c7db16-1364-477c-8b49-489562b8683c-serviceca\") pod \"node-ca-x54gh\" (UID: \"c7c7db16-1364-477c-8b49-489562b8683c\") " pod="openshift-image-registry/node-ca-x54gh"
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.051706    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs podName:e49d21df-0b01-4845-ad69-b2ad83b93187 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:47.55169124 +0000 UTC m=+2.959280927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs") pod "network-metrics-daemon-wcsrt" (UID: "e49d21df-0b01-4845-ad69-b2ad83b93187") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:47.051718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051721    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fd2acb5-01d8-40e2-9ffc-729199b598d0-agent-certs\") pod \"konnectivity-agent-q27h5\" (UID: \"4fd2acb5-01d8-40e2-9ffc-729199b598d0\") " pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051742    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-system-cni-dir\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051769    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-system-cni-dir\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051792    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-lib-modules\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051776    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b740e4f-f613-4235-8aa1-ec671ed8208a-cni-binary-copy\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051823    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051847    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-var-lib-kubelet\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051870    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-sysconfig\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051886    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/785f14dc-4626-4437-be21-2761e326fe7d-tmp-dir\") pod \"node-resolver-h4v89\" (UID: \"785f14dc-4626-4437-be21-2761e326fe7d\") " pod="openshift-dns/node-resolver-h4v89"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051894    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-socket-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051929    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b89780e-214b-4029-8ee8-386e60ec5437-host-slash\") pod \"iptables-alerter-7n4fq\" (UID: \"0b89780e-214b-4029-8ee8-386e60ec5437\") " pod="openshift-network-operator/iptables-alerter-7n4fq"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051952    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b740e4f-f613-4235-8aa1-ec671ed8208a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051973    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-sysconfig\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051974    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/549d5421-b20a-4288-88c3-c3629f566982-cni-binary-copy\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.051995    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052008    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-device-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052017    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-var-lib-kubelet\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.052168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052046    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b89780e-214b-4029-8ee8-386e60ec5437-host-slash\") pod \"iptables-alerter-7n4fq\" (UID: \"0b89780e-214b-4029-8ee8-386e60ec5437\") " pod="openshift-network-operator/iptables-alerter-7n4fq"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052167    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-kubelet\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052210    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrkd\" (UniqueName: \"kubernetes.io/projected/e49d21df-0b01-4845-ad69-b2ad83b93187-kube-api-access-rkrkd\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-multus-conf-dir\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052263    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-sysctl-d\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052309    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-systemd\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052323    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-tuned\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052326    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b740e4f-f613-4235-8aa1-ec671ed8208a-cni-binary-copy\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052360    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjgkv\" (UniqueName: \"kubernetes.io/projected/abebe566-c814-44bf-a8a0-c091d6d73fb4-kube-api-access-mjgkv\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052425    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7c7db16-1364-477c-8b49-489562b8683c-host\") pod \"node-ca-x54gh\" (UID: \"c7c7db16-1364-477c-8b49-489562b8683c\") " pod="openshift-image-registry/node-ca-x54gh"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052467    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/785f14dc-4626-4437-be21-2761e326fe7d-hosts-file\") pod \"node-resolver-h4v89\" (UID: \"785f14dc-4626-4437-be21-2761e326fe7d\") " pod="openshift-dns/node-resolver-h4v89"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052476    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b740e4f-f613-4235-8aa1-ec671ed8208a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052491    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ndsr\" (UniqueName: \"kubernetes.io/projected/549d5421-b20a-4288-88c3-c3629f566982-kube-api-access-9ndsr\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052539    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97b6993b-5c3c-400b-85fd-2044c0ae3eff-tmp\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052544    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/549d5421-b20a-4288-88c3-c3629f566982-cni-binary-copy\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052565    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-run-systemd\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052588    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-run-openvswitch\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052591    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-multus-conf-dir\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.052801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052608    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052658    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-systemd\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052693    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/785f14dc-4626-4437-be21-2761e326fe7d-hosts-file\") pod \"node-resolver-h4v89\" (UID: \"785f14dc-4626-4437-be21-2761e326fe7d\") " pod="openshift-dns/node-resolver-h4v89"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052694    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-sysctl-d\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052610    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/818a88fe-5429-40b4-998f-cd34b2a149d8-dbus\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052802    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052828    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-etc-kubernetes\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052848    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-etc-selinux\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052870    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-etc-openvswitch\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052896    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859nm\" (UniqueName: \"kubernetes.io/projected/68468352-f49f-497c-b807-a5e0d7e50082-kube-api-access-859nm\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052907    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-etc-kubernetes\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052922    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b89780e-214b-4029-8ee8-386e60ec5437-iptables-alerter-script\") pod \"iptables-alerter-7n4fq\" (UID: \"0b89780e-214b-4029-8ee8-386e60ec5437\") " pod="openshift-network-operator/iptables-alerter-7n4fq"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052945    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-run\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052966    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-systemd-units\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.052987    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-var-lib-openvswitch\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053007    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-cni-netd\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053029    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4mdp\" (UniqueName: \"kubernetes.io/projected/785f14dc-4626-4437-be21-2761e326fe7d-kube-api-access-s4mdp\") pod \"node-resolver-h4v89\" (UID: \"785f14dc-4626-4437-be21-2761e326fe7d\") " pod="openshift-dns/node-resolver-h4v89"
Jul 07 02:17:47.053430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053059    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b740e4f-f613-4235-8aa1-ec671ed8208a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053100    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmptk\" (UniqueName: \"kubernetes.io/projected/c7c7db16-1364-477c-8b49-489562b8683c-kube-api-access-pmptk\") pod \"node-ca-x54gh\" (UID: \"c7c7db16-1364-477c-8b49-489562b8683c\") " pod="openshift-image-registry/node-ca-x54gh"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053123    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-slash\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053147    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-log-socket\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053168    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053194    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68468352-f49f-497c-b807-a5e0d7e50082-ovnkube-script-lib\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053216    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/818a88fe-5429-40b4-998f-cd34b2a149d8-kubelet-config\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053238    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-system-cni-dir\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053271    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-multus-socket-dir-parent\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053292    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-host\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053313    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053333    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-sys-fs\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053383    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68468352-f49f-497c-b807-a5e0d7e50082-ovn-node-metrics-cert\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053406    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg92v\" (UniqueName: \"kubernetes.io/projected/2b740e4f-f613-4235-8aa1-ec671ed8208a-kube-api-access-wg92v\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053429    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-cnibin\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053461    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-var-lib-cni-multus\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053405    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b89780e-214b-4029-8ee8-386e60ec5437-iptables-alerter-script\") pod \"iptables-alerter-7n4fq\" (UID: \"0b89780e-214b-4029-8ee8-386e60ec5437\") " pod="openshift-network-operator/iptables-alerter-7n4fq"
Jul 07 02:17:47.054034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053541    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-system-cni-dir\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053583    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-multus-socket-dir-parent\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053694    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-cnibin\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053745    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-run\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053768    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-run-multus-certs\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053783    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-host\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053793    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-sys\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053817    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-run-netns\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053831    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-var-lib-cni-multus\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053842    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-multus-cni-dir\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053876    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-hostroot\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053893    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-sys\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053904    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-run-ovn\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053924    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b740e4f-f613-4235-8aa1-ec671ed8208a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053929    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/549d5421-b20a-4288-88c3-c3629f566982-multus-daemon-config\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053974    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-cni-bin\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053997    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-multus-cni-dir\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.053998    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fd2acb5-01d8-40e2-9ffc-729199b598d0-konnectivity-ca\") pod \"konnectivity-agent-q27h5\" (UID: \"4fd2acb5-01d8-40e2-9ffc-729199b598d0\") " pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:17:47.054730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054027    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-cnibin\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054071    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-hostroot\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054097    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-sysctl-conf\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054104    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-cnibin\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054124    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-var-lib-kubelet\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054157    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-registration-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054187    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-node-log\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054209    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68468352-f49f-497c-b807-a5e0d7e50082-ovnkube-config\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054206    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-var-lib-kubelet\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsznz\" (UniqueName: \"kubernetes.io/projected/0b89780e-214b-4029-8ee8-386e60ec5437-kube-api-access-wsznz\") pod \"iptables-alerter-7n4fq\" (UID: \"0b89780e-214b-4029-8ee8-386e60ec5437\") " pod="openshift-network-operator/iptables-alerter-7n4fq"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054254    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-sysctl-conf\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054277    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-os-release\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054293    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-run-multus-certs\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054321    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/549d5421-b20a-4288-88c3-c3629f566982-multus-daemon-config\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054339    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-run-netns\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054392    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-modprobe-d\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054412    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2b740e4f-f613-4235-8aa1-ec671ed8208a-os-release\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.055311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054440    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/549d5421-b20a-4288-88c3-c3629f566982-host-run-netns\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.055927 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054453    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-kubernetes\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.055927 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054518    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-modprobe-d\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.055927 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.054556    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-kubernetes\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.055927 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.055817    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97b6993b-5c3c-400b-85fd-2044c0ae3eff-tmp\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.056400 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.056381    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97b6993b-5c3c-400b-85fd-2044c0ae3eff-etc-tuned\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.059686 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.059666    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcblw\" (UniqueName: \"kubernetes.io/projected/97b6993b-5c3c-400b-85fd-2044c0ae3eff-kube-api-access-lcblw\") pod \"tuned-vsdng\" (UID: \"97b6993b-5c3c-400b-85fd-2044c0ae3eff\") " pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.060383 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.060361    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ndsr\" (UniqueName: \"kubernetes.io/projected/549d5421-b20a-4288-88c3-c3629f566982-kube-api-access-9ndsr\") pod \"multus-vdkb4\" (UID: \"549d5421-b20a-4288-88c3-c3629f566982\") " pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.060628 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.060608    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrkd\" (UniqueName: \"kubernetes.io/projected/e49d21df-0b01-4845-ad69-b2ad83b93187-kube-api-access-rkrkd\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:47.061075 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.061057    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg92v\" (UniqueName: \"kubernetes.io/projected/2b740e4f-f613-4235-8aa1-ec671ed8208a-kube-api-access-wg92v\") pod \"multus-additional-cni-plugins-sk7b9\" (UID: \"2b740e4f-f613-4235-8aa1-ec671ed8208a\") " pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.061528 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.061514    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4mdp\" (UniqueName: \"kubernetes.io/projected/785f14dc-4626-4437-be21-2761e326fe7d-kube-api-access-s4mdp\") pod \"node-resolver-h4v89\" (UID: \"785f14dc-4626-4437-be21-2761e326fe7d\") " pod="openshift-dns/node-resolver-h4v89"
Jul 07 02:17:47.061861 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.061847    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsznz\" (UniqueName: \"kubernetes.io/projected/0b89780e-214b-4029-8ee8-386e60ec5437-kube-api-access-wsznz\") pod \"iptables-alerter-7n4fq\" (UID: \"0b89780e-214b-4029-8ee8-386e60ec5437\") " pod="openshift-network-operator/iptables-alerter-7n4fq"
Jul 07 02:17:47.086578 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.086522    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal" event={"ID":"dd0c9e68fa1554fd01092c3bef64ba0f","Type":"ContainerStarted","Data":"d4a3960a06a1c7f64b24f6ea7802d31264ff37dd98a825107e94cb7564e4e57d"}
Jul 07 02:17:47.087594 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.087573    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal" event={"ID":"bd320ee8d7c30c12d7b97d56aa7806c5","Type":"ContainerStarted","Data":"bd0b905e3ef93cdd5bb5e4306c05c380ea1ad89d3ebef1162f80a52fccbad9c2"}
Jul 07 02:17:47.155308 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155247    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.155308 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155279    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-sys-fs\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.155308 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155301    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68468352-f49f-497c-b807-a5e0d7e50082-ovn-node-metrics-cert\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155335    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-run-netns\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155388    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.155539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155388    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-sys-fs\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.155539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155427    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-run-ovn\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155439    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-run-netns\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155445    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-cni-bin\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155466    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-run-ovn\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155466    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fd2acb5-01d8-40e2-9ffc-729199b598d0-konnectivity-ca\") pod \"konnectivity-agent-q27h5\" (UID: \"4fd2acb5-01d8-40e2-9ffc-729199b598d0\") " pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:17:47.155539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155500    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-registration-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.155539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155514    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-node-log\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155544    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68468352-f49f-497c-b807-a5e0d7e50082-ovnkube-config\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155574    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-node-log\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155581    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155605    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68468352-f49f-497c-b807-a5e0d7e50082-env-overrides\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155642    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdg6\" (UniqueName: \"kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6\") pod \"network-check-target-7sr2n\" (UID: \"c2b43d9f-097e-4e87-a0a6-40546d9005fe\") " pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155605    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-cni-bin\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155664    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7c7db16-1364-477c-8b49-489562b8683c-serviceca\") pod \"node-ca-x54gh\" (UID: \"c7c7db16-1364-477c-8b49-489562b8683c\") " pod="openshift-image-registry/node-ca-x54gh"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155684    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fd2acb5-01d8-40e2-9ffc-729199b598d0-agent-certs\") pod \"konnectivity-agent-q27h5\" (UID: \"4fd2acb5-01d8-40e2-9ffc-729199b598d0\") " pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155682    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155643    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-registration-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155712    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-socket-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155750    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-device-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155782    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-kubelet\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155811    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjgkv\" (UniqueName: \"kubernetes.io/projected/abebe566-c814-44bf-a8a0-c091d6d73fb4-kube-api-access-mjgkv\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155833    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7c7db16-1364-477c-8b49-489562b8683c-host\") pod \"node-ca-x54gh\" (UID: \"c7c7db16-1364-477c-8b49-489562b8683c\") " pod="openshift-image-registry/node-ca-x54gh"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155856    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-run-systemd\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.155954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155877    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-run-openvswitch\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155898    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/818a88fe-5429-40b4-998f-cd34b2a149d8-dbus\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155917    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155939    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4fd2acb5-01d8-40e2-9ffc-729199b598d0-konnectivity-ca\") pod \"konnectivity-agent-q27h5\" (UID: \"4fd2acb5-01d8-40e2-9ffc-729199b598d0\") " pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155940    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-etc-selinux\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155975    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-etc-openvswitch\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.155994    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859nm\" (UniqueName: \"kubernetes.io/projected/68468352-f49f-497c-b807-a5e0d7e50082-kube-api-access-859nm\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156000    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-socket-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156017    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-systemd-units\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156024    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-etc-selinux\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156040    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-var-lib-openvswitch\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156050    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7c7db16-1364-477c-8b49-489562b8683c-host\") pod \"node-ca-x54gh\" (UID: \"c7c7db16-1364-477c-8b49-489562b8683c\") " pod="openshift-image-registry/node-ca-x54gh"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156059    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-run-systemd\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156062    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-cni-netd\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156085    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmptk\" (UniqueName: \"kubernetes.io/projected/c7c7db16-1364-477c-8b49-489562b8683c-kube-api-access-pmptk\") pod \"node-ca-x54gh\" (UID: \"c7c7db16-1364-477c-8b49-489562b8683c\") " pod="openshift-image-registry/node-ca-x54gh"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156093    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7c7db16-1364-477c-8b49-489562b8683c-serviceca\") pod \"node-ca-x54gh\" (UID: \"c7c7db16-1364-477c-8b49-489562b8683c\") " pod="openshift-image-registry/node-ca-x54gh"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156102    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-kubelet\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.156731 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156110    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-slash\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-log-socket\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.156136    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156167    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156193    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68468352-f49f-497c-b807-a5e0d7e50082-ovnkube-script-lib\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.156202    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret podName:818a88fe-5429-40b4-998f-cd34b2a149d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:47.6561876 +0000 UTC m=+3.063777281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret") pod "global-pull-secret-syncer-pg4n4" (UID: "818a88fe-5429-40b4-998f-cd34b2a149d8") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156210    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/818a88fe-5429-40b4-998f-cd34b2a149d8-dbus\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156225    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/818a88fe-5429-40b4-998f-cd34b2a149d8-kubelet-config\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156138    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-var-lib-openvswitch\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156296    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-systemd-units\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156365    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/818a88fe-5429-40b4-998f-cd34b2a149d8-kubelet-config\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156058    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/68468352-f49f-497c-b807-a5e0d7e50082-env-overrides\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156418    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-cni-netd\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156422    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/68468352-f49f-497c-b807-a5e0d7e50082-ovnkube-config\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156460    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-etc-openvswitch\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156469    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-slash\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156094    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abebe566-c814-44bf-a8a0-c091d6d73fb4-device-dir\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.157251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156493    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-log-socket\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157931 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156510    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157931 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156516    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/68468352-f49f-497c-b807-a5e0d7e50082-run-openvswitch\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.157931 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.156626    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/68468352-f49f-497c-b807-a5e0d7e50082-ovnkube-script-lib\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.158110 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.158089    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/68468352-f49f-497c-b807-a5e0d7e50082-ovn-node-metrics-cert\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.158319 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.158305    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4fd2acb5-01d8-40e2-9ffc-729199b598d0-agent-certs\") pod \"konnectivity-agent-q27h5\" (UID: \"4fd2acb5-01d8-40e2-9ffc-729199b598d0\") " pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:17:47.165335 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.165315    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:17:47.165335 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.165359    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:17:47.165499 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.165373    2570 projected.go:196] Error preparing data for projected volume kube-api-access-7hdg6 for pod openshift-network-diagnostics/network-check-target-7sr2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:47.165548 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.165532    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6 podName:c2b43d9f-097e-4e87-a0a6-40546d9005fe nodeName:}" failed. No retries permitted until 2026-07-07 02:17:47.665472035 +0000 UTC m=+3.073061700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7hdg6" (UniqueName: "kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6") pod "network-check-target-7sr2n" (UID: "c2b43d9f-097e-4e87-a0a6-40546d9005fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:47.166771 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.166750    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmptk\" (UniqueName: \"kubernetes.io/projected/c7c7db16-1364-477c-8b49-489562b8683c-kube-api-access-pmptk\") pod \"node-ca-x54gh\" (UID: \"c7c7db16-1364-477c-8b49-489562b8683c\") " pod="openshift-image-registry/node-ca-x54gh"
Jul 07 02:17:47.167101 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.167083    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-859nm\" (UniqueName: \"kubernetes.io/projected/68468352-f49f-497c-b807-a5e0d7e50082-kube-api-access-859nm\") pod \"ovnkube-node-vcklb\" (UID: \"68468352-f49f-497c-b807-a5e0d7e50082\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.167254 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.167239    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjgkv\" (UniqueName: \"kubernetes.io/projected/abebe566-c814-44bf-a8a0-c091d6d73fb4-kube-api-access-mjgkv\") pod \"aws-ebs-csi-driver-node-cvpmr\" (UID: \"abebe566-c814-44bf-a8a0-c091d6d73fb4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.241416 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.241389    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7n4fq"
Jul 07 02:17:47.247330 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.247316    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h4v89"
Jul 07 02:17:47.257676 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.257659    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sk7b9"
Jul 07 02:17:47.265144 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.265126    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vdkb4"
Jul 07 02:17:47.271182 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.271169    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vsdng"
Jul 07 02:17:47.277428 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.277407    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x54gh"
Jul 07 02:17:47.283824 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.283810    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:17:47.291214 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.291187    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:17:47.296697 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.296685    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr"
Jul 07 02:17:47.558807 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.558733    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:47.558953 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.558865    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:47.558953 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.558931    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs podName:e49d21df-0b01-4845-ad69-b2ad83b93187 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:48.558912357 +0000 UTC m=+3.966502023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs") pod "network-metrics-daemon-wcsrt" (UID: "e49d21df-0b01-4845-ad69-b2ad83b93187") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:47.659713 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.659677    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:47.659846 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.659799    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:47.659913 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.659872    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret podName:818a88fe-5429-40b4-998f-cd34b2a149d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:48.659851784 +0000 UTC m=+4.067441453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret") pod "global-pull-secret-syncer-pg4n4" (UID: "818a88fe-5429-40b4-998f-cd34b2a149d8") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:47.760788 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.760757    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdg6\" (UniqueName: \"kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6\") pod \"network-check-target-7sr2n\" (UID: \"c2b43d9f-097e-4e87-a0a6-40546d9005fe\") " pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:47.760947 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.760924    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:17:47.760947 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.760945    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:17:47.761050 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.760957    2570 projected.go:196] Error preparing data for projected volume kube-api-access-7hdg6 for pod openshift-network-diagnostics/network-check-target-7sr2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:47.761050 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:47.761032    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6 podName:c2b43d9f-097e-4e87-a0a6-40546d9005fe nodeName:}" failed. No retries permitted until 2026-07-07 02:17:48.761000903 +0000 UTC m=+4.168590568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7hdg6" (UniqueName: "kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6") pod "network-check-target-7sr2n" (UID: "c2b43d9f-097e-4e87-a0a6-40546d9005fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:47.955246 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:47.955221    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b89780e_214b_4029_8ee8_386e60ec5437.slice/crio-e10f4785b9a5d865ad60bbec60a633312979a35a65af6f15677f016ab19d3015 WatchSource:0}: Error finding container e10f4785b9a5d865ad60bbec60a633312979a35a65af6f15677f016ab19d3015: Status 404 returned error can't find the container with id e10f4785b9a5d865ad60bbec60a633312979a35a65af6f15677f016ab19d3015
Jul 07 02:17:47.956979 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:47.956957    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c7db16_1364_477c_8b49_489562b8683c.slice/crio-0def2e94810baeaed1b08203954021c69055011ccb31e634a614d80d86c4a4fd WatchSource:0}: Error finding container 0def2e94810baeaed1b08203954021c69055011ccb31e634a614d80d86c4a4fd: Status 404 returned error can't find the container with id 0def2e94810baeaed1b08203954021c69055011ccb31e634a614d80d86c4a4fd
Jul 07 02:17:47.957613 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:47.957587    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabebe566_c814_44bf_a8a0_c091d6d73fb4.slice/crio-9cbd6e73f7c708352c01f8aac524b99ad86c755679c063c178745195fbc4e257 WatchSource:0}: Error finding container 9cbd6e73f7c708352c01f8aac524b99ad86c755679c063c178745195fbc4e257: Status 404 returned error can't find the container with id 9cbd6e73f7c708352c01f8aac524b99ad86c755679c063c178745195fbc4e257
Jul 07 02:17:47.958387 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:47.958280    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785f14dc_4626_4437_be21_2761e326fe7d.slice/crio-053e77c32aeba82eec4315b7cfa9861e2b356ef505c41ae82a7f74770560f0d0 WatchSource:0}: Error finding container 053e77c32aeba82eec4315b7cfa9861e2b356ef505c41ae82a7f74770560f0d0: Status 404 returned error can't find the container with id 053e77c32aeba82eec4315b7cfa9861e2b356ef505c41ae82a7f74770560f0d0
Jul 07 02:17:47.961255 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:47.961241    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549d5421_b20a_4288_88c3_c3629f566982.slice/crio-a470b2f01747b176939b4376f5f5381fdef9f9da2437fbf2fc95cc040c56030e WatchSource:0}: Error finding container a470b2f01747b176939b4376f5f5381fdef9f9da2437fbf2fc95cc040c56030e: Status 404 returned error can't find the container with id a470b2f01747b176939b4376f5f5381fdef9f9da2437fbf2fc95cc040c56030e
Jul 07 02:17:47.962030 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:47.962015    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68468352_f49f_497c_b807_a5e0d7e50082.slice/crio-6a094123af8bae5cef70697e2135399f933318d02710891f9fd6a6db46c606b5 WatchSource:0}: Error finding container 6a094123af8bae5cef70697e2135399f933318d02710891f9fd6a6db46c606b5: Status 404 returned error can't find the container with id 6a094123af8bae5cef70697e2135399f933318d02710891f9fd6a6db46c606b5
Jul 07 02:17:47.963256 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:47.963237    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b6993b_5c3c_400b_85fd_2044c0ae3eff.slice/crio-79254953b11994774d3be364a24b6be8e6cea3a0a628fb8f84b607b259b9d0fb WatchSource:0}: Error finding container 79254953b11994774d3be364a24b6be8e6cea3a0a628fb8f84b607b259b9d0fb: Status 404 returned error can't find the container with id 79254953b11994774d3be364a24b6be8e6cea3a0a628fb8f84b607b259b9d0fb
Jul 07 02:17:47.965044 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:47.965022    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd2acb5_01d8_40e2_9ffc_729199b598d0.slice/crio-47d410d0df643d1a8effef74e418d1420411251d44c87933b3368f7bcd155096 WatchSource:0}: Error finding container 47d410d0df643d1a8effef74e418d1420411251d44c87933b3368f7bcd155096: Status 404 returned error can't find the container with id 47d410d0df643d1a8effef74e418d1420411251d44c87933b3368f7bcd155096
Jul 07 02:17:47.966972 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:17:47.966701    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b740e4f_f613_4235_8aa1_ec671ed8208a.slice/crio-fbdb30e8233345f4762e0f040a891e29d4602cc0e418bf61d7d527cac578fb7d WatchSource:0}: Error finding container fbdb30e8233345f4762e0f040a891e29d4602cc0e418bf61d7d527cac578fb7d: Status 404 returned error can't find the container with id fbdb30e8233345f4762e0f040a891e29d4602cc0e418bf61d7d527cac578fb7d
Jul 07 02:17:47.985244 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.985215    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:12:46 +0000 UTC" deadline="2028-02-26 22:51:07.338761292 +0000 UTC"
Jul 07 02:17:47.985244 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:47.985244    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14396h33m19.353521206s"
Jul 07 02:17:48.081666 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.081541    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:48.081751 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:48.081737    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:17:48.090925 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.090903    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h4v89" event={"ID":"785f14dc-4626-4437-be21-2761e326fe7d","Type":"ContainerStarted","Data":"053e77c32aeba82eec4315b7cfa9861e2b356ef505c41ae82a7f74770560f0d0"}
Jul 07 02:17:48.091878 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.091852    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x54gh" event={"ID":"c7c7db16-1364-477c-8b49-489562b8683c","Type":"ContainerStarted","Data":"0def2e94810baeaed1b08203954021c69055011ccb31e634a614d80d86c4a4fd"}
Jul 07 02:17:48.095070 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.093143    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7n4fq" event={"ID":"0b89780e-214b-4029-8ee8-386e60ec5437","Type":"ContainerStarted","Data":"e10f4785b9a5d865ad60bbec60a633312979a35a65af6f15677f016ab19d3015"}
Jul 07 02:17:48.096510 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.096486    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q27h5" event={"ID":"4fd2acb5-01d8-40e2-9ffc-729199b598d0","Type":"ContainerStarted","Data":"47d410d0df643d1a8effef74e418d1420411251d44c87933b3368f7bcd155096"}
Jul 07 02:17:48.097428 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.097405    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb" event={"ID":"68468352-f49f-497c-b807-a5e0d7e50082","Type":"ContainerStarted","Data":"6a094123af8bae5cef70697e2135399f933318d02710891f9fd6a6db46c606b5"}
Jul 07 02:17:48.099884 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.099859    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal" event={"ID":"dd0c9e68fa1554fd01092c3bef64ba0f","Type":"ContainerStarted","Data":"f0779ae38b487d6e73b83fd4eaa91b35a94162c5cfe825657d27502acbf4a32f"}
Jul 07 02:17:48.100802 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.100784    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sk7b9" event={"ID":"2b740e4f-f613-4235-8aa1-ec671ed8208a","Type":"ContainerStarted","Data":"fbdb30e8233345f4762e0f040a891e29d4602cc0e418bf61d7d527cac578fb7d"}
Jul 07 02:17:48.102927 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.102908    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vsdng" event={"ID":"97b6993b-5c3c-400b-85fd-2044c0ae3eff","Type":"ContainerStarted","Data":"79254953b11994774d3be364a24b6be8e6cea3a0a628fb8f84b607b259b9d0fb"}
Jul 07 02:17:48.104003 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.103979    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr" event={"ID":"abebe566-c814-44bf-a8a0-c091d6d73fb4","Type":"ContainerStarted","Data":"9cbd6e73f7c708352c01f8aac524b99ad86c755679c063c178745195fbc4e257"}
Jul 07 02:17:48.105373 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.105331    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vdkb4" event={"ID":"549d5421-b20a-4288-88c3-c3629f566982","Type":"ContainerStarted","Data":"a470b2f01747b176939b4376f5f5381fdef9f9da2437fbf2fc95cc040c56030e"}
Jul 07 02:17:48.113551 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.113517    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal" podStartSLOduration=3.113506787 podStartE2EDuration="3.113506787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:17:48.113116201 +0000 UTC m=+3.520705885" watchObservedRunningTime="2026-07-07 02:17:48.113506787 +0000 UTC m=+3.521096471"
Jul 07 02:17:48.565730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.565644    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:48.565891 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:48.565774    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:48.565891 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:48.565834    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs podName:e49d21df-0b01-4845-ad69-b2ad83b93187 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:50.565815789 +0000 UTC m=+5.973405463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs") pod "network-metrics-daemon-wcsrt" (UID: "e49d21df-0b01-4845-ad69-b2ad83b93187") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:48.666504 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.666464    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:48.666663 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:48.666627    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:48.666738 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:48.666696    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret podName:818a88fe-5429-40b4-998f-cd34b2a149d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:50.666677003 +0000 UTC m=+6.074266681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret") pod "global-pull-secret-syncer-pg4n4" (UID: "818a88fe-5429-40b4-998f-cd34b2a149d8") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:48.767164 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.767130    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdg6\" (UniqueName: \"kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6\") pod \"network-check-target-7sr2n\" (UID: \"c2b43d9f-097e-4e87-a0a6-40546d9005fe\") " pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:48.767320 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:48.767295    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:17:48.767320 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:48.767316    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:17:48.767473 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:48.767329    2570 projected.go:196] Error preparing data for projected volume kube-api-access-7hdg6 for pod openshift-network-diagnostics/network-check-target-7sr2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:48.767473 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:48.767403    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6 podName:c2b43d9f-097e-4e87-a0a6-40546d9005fe nodeName:}" failed. No retries permitted until 2026-07-07 02:17:50.767384782 +0000 UTC m=+6.174974448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7hdg6" (UniqueName: "kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6") pod "network-check-target-7sr2n" (UID: "c2b43d9f-097e-4e87-a0a6-40546d9005fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:48.986456 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.986314    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:12:46 +0000 UTC" deadline="2028-04-19 07:03:18.343978064 +0000 UTC"
Jul 07 02:17:48.986456 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:48.986405    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15652h45m29.357577179s"
Jul 07 02:17:49.083820 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:49.083789    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:49.083997 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:49.083928    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:17:49.084336 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:49.084317    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:49.084454 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:49.084433    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:17:49.117569 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:49.117516    2570 generic.go:356] "Generic (PLEG): container finished" podID="bd320ee8d7c30c12d7b97d56aa7806c5" containerID="f2d0f1eae265dcb18b0a180fc2f79e0fc1d261413204a955e48484c0f10e9006" exitCode=0
Jul 07 02:17:49.117707 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:49.117684    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal" event={"ID":"bd320ee8d7c30c12d7b97d56aa7806c5","Type":"ContainerDied","Data":"f2d0f1eae265dcb18b0a180fc2f79e0fc1d261413204a955e48484c0f10e9006"}
Jul 07 02:17:50.081114 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:50.081086    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:50.081595 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:50.081207    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:17:50.124213 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:50.124141    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal" event={"ID":"bd320ee8d7c30c12d7b97d56aa7806c5","Type":"ContainerStarted","Data":"05ee49117268a047061cb220e5790b8745dfe2b6d8f80384da0dc54341ae3c23"}
Jul 07 02:17:50.582290 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:50.582257    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:50.582506 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:50.582488    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:50.582571 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:50.582564    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs podName:e49d21df-0b01-4845-ad69-b2ad83b93187 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:54.582545761 +0000 UTC m=+9.990135449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs") pod "network-metrics-daemon-wcsrt" (UID: "e49d21df-0b01-4845-ad69-b2ad83b93187") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:50.683621 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:50.683587    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:50.683786 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:50.683752    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:50.683847 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:50.683812    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret podName:818a88fe-5429-40b4-998f-cd34b2a149d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:17:54.683795236 +0000 UTC m=+10.091384902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret") pod "global-pull-secret-syncer-pg4n4" (UID: "818a88fe-5429-40b4-998f-cd34b2a149d8") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:50.784835 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:50.784252    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdg6\" (UniqueName: \"kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6\") pod \"network-check-target-7sr2n\" (UID: \"c2b43d9f-097e-4e87-a0a6-40546d9005fe\") " pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:50.784835 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:50.784415    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:17:50.784835 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:50.784437    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:17:50.784835 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:50.784449    2570 projected.go:196] Error preparing data for projected volume kube-api-access-7hdg6 for pod openshift-network-diagnostics/network-check-target-7sr2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:50.784835 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:50.784504    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6 podName:c2b43d9f-097e-4e87-a0a6-40546d9005fe nodeName:}" failed. No retries permitted until 2026-07-07 02:17:54.78448709 +0000 UTC m=+10.192076757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7hdg6" (UniqueName: "kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6") pod "network-check-target-7sr2n" (UID: "c2b43d9f-097e-4e87-a0a6-40546d9005fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:51.080735 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:51.080657    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:51.080891 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:51.080788    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:17:51.080891 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:51.080843    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:51.080997 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:51.080955    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:17:52.081312 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:52.081279    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:52.081861 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:52.081389    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:17:53.081492 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:53.081454    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:53.081942 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:53.081595    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:17:53.081942 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:53.081938    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:53.082075 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:53.082031    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:17:54.080900 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:54.080821    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:54.081034 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:54.080937    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:17:54.615314 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:54.615283    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:54.615730 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:54.615435    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:54.615730 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:54.615495    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs podName:e49d21df-0b01-4845-ad69-b2ad83b93187 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:02.615475873 +0000 UTC m=+18.023065542 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs") pod "network-metrics-daemon-wcsrt" (UID: "e49d21df-0b01-4845-ad69-b2ad83b93187") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:17:54.715732 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:54.715699    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:54.715919 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:54.715845    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:54.715919 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:54.715912    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret podName:818a88fe-5429-40b4-998f-cd34b2a149d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:02.715895544 +0000 UTC m=+18.123485226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret") pod "global-pull-secret-syncer-pg4n4" (UID: "818a88fe-5429-40b4-998f-cd34b2a149d8") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:17:54.817107 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:54.816537    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdg6\" (UniqueName: \"kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6\") pod \"network-check-target-7sr2n\" (UID: \"c2b43d9f-097e-4e87-a0a6-40546d9005fe\") " pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:54.817107 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:54.816710    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:17:54.817107 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:54.816732    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:17:54.817107 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:54.816742    2570 projected.go:196] Error preparing data for projected volume kube-api-access-7hdg6 for pod openshift-network-diagnostics/network-check-target-7sr2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:54.817107 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:54.816793    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6 podName:c2b43d9f-097e-4e87-a0a6-40546d9005fe nodeName:}" failed. No retries permitted until 2026-07-07 02:18:02.816774449 +0000 UTC m=+18.224364117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7hdg6" (UniqueName: "kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6") pod "network-check-target-7sr2n" (UID: "c2b43d9f-097e-4e87-a0a6-40546d9005fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:17:55.082166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:55.082098    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:55.082315 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:55.082210    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:17:55.082703 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:55.082679    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:55.082840 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:55.082799    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:17:56.081195 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:56.081165    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:56.081603 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:56.081298    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:17:57.081502 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:57.081468    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:57.081965 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:57.081605    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:17:57.082022 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:57.081996    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:57.082144 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:57.082083    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:17:58.081214 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:58.081182    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:17:58.081408 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:58.081292    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:17:59.083927 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:59.083898    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:17:59.084321 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:59.084015    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:17:59.084321 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:17:59.084053    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:17:59.084321 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:17:59.084114    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:00.081503 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:00.081472    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:00.081691 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:00.081586    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:18:01.083493 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:01.083463    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:01.083888 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:01.083473    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:01.083888 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:01.083562    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:18:01.083888 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:01.083644    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:02.080684 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:02.080652    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:02.080866 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:02.080764    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:18:02.675997 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:02.675968    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:02.676360 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:02.676069    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:02.676360 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:02.676118    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs podName:e49d21df-0b01-4845-ad69-b2ad83b93187 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:18.67610299 +0000 UTC m=+34.083692668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs") pod "network-metrics-daemon-wcsrt" (UID: "e49d21df-0b01-4845-ad69-b2ad83b93187") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:02.776441 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:02.776406    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:02.776578 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:02.776556    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:02.776625 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:02.776618    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret podName:818a88fe-5429-40b4-998f-cd34b2a149d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:18.776604047 +0000 UTC m=+34.184193715 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret") pod "global-pull-secret-syncer-pg4n4" (UID: "818a88fe-5429-40b4-998f-cd34b2a149d8") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:02.876938 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:02.876898    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdg6\" (UniqueName: \"kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6\") pod \"network-check-target-7sr2n\" (UID: \"c2b43d9f-097e-4e87-a0a6-40546d9005fe\") " pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:02.877105 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:02.877077    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:02.877105 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:02.877098    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:02.877198 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:02.877108    2570 projected.go:196] Error preparing data for projected volume kube-api-access-7hdg6 for pod openshift-network-diagnostics/network-check-target-7sr2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:02.877198 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:02.877164    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6 podName:c2b43d9f-097e-4e87-a0a6-40546d9005fe nodeName:}" failed. No retries permitted until 2026-07-07 02:18:18.87715074 +0000 UTC m=+34.284740408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7hdg6" (UniqueName: "kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6") pod "network-check-target-7sr2n" (UID: "c2b43d9f-097e-4e87-a0a6-40546d9005fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:03.081299 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:03.081224    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:03.081474 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:03.081235    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:03.081474 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:03.081362    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:03.081474 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:03.081431    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:18:04.081569 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:04.081539    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:04.081992 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:04.081659    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:18:05.082279 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:05.082246    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:05.082731 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:05.082372    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:05.082895 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:05.082867    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:05.083018 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:05.082967    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:18:06.081280 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:06.081251    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:06.081460 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:06.081383    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:18:07.081178 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:07.081150    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:07.081464 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:07.081150    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:07.081464 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:07.081275    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:07.081464 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:07.081321    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:18:08.081623 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.081229    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:08.082316 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:08.081698    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:18:08.159411 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.159377    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h4v89" event={"ID":"785f14dc-4626-4437-be21-2761e326fe7d","Type":"ContainerStarted","Data":"8c07b0b20f93767b93d9894a2bb0425547f35ac2dc1db1796dc94fe1195d5a39"}
Jul 07 02:18:08.160772 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.160747    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x54gh" event={"ID":"c7c7db16-1364-477c-8b49-489562b8683c","Type":"ContainerStarted","Data":"a5b3c7098a904e2bbcc762b4f5e09e5280ff025b0d4a3da4c914b282d33010e5"}
Jul 07 02:18:08.162092 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.162067    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q27h5" event={"ID":"4fd2acb5-01d8-40e2-9ffc-729199b598d0","Type":"ContainerStarted","Data":"5449816e82c541aca78470440bdd8836cf587267aa708fa4342264cb864a243b"}
Jul 07 02:18:08.164637 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.164617    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb" event={"ID":"68468352-f49f-497c-b807-a5e0d7e50082","Type":"ContainerStarted","Data":"f20650beaa55d312772b27bfebb87818e885bc5dae6d3e34fb3dcd8d9fcd9426"}
Jul 07 02:18:08.164730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.164667    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb" event={"ID":"68468352-f49f-497c-b807-a5e0d7e50082","Type":"ContainerStarted","Data":"7dfebaa7951b2437f2120ff251e9793ed409988b55b5e65e6bf13f707a3a4ce8"}
Jul 07 02:18:08.164730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.164696    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb" event={"ID":"68468352-f49f-497c-b807-a5e0d7e50082","Type":"ContainerStarted","Data":"5a59a5f6915ade5790262aecb9f2ffabe31a3ea176fd6b66d881c53b619eb492"}
Jul 07 02:18:08.164730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.164724    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb" event={"ID":"68468352-f49f-497c-b807-a5e0d7e50082","Type":"ContainerStarted","Data":"ae578e1126e3ff70d7dff3b5ad534454cc1cc17af5f67184cdec0465e7cfb45b"}
Jul 07 02:18:08.164851 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.164750    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb" event={"ID":"68468352-f49f-497c-b807-a5e0d7e50082","Type":"ContainerStarted","Data":"adc9eaba2735f3bf9f2d5b87ea874ac4f99f711cc8ce80ac0ce6aaa09bee5ead"}
Jul 07 02:18:08.164851 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.164777    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb" event={"ID":"68468352-f49f-497c-b807-a5e0d7e50082","Type":"ContainerStarted","Data":"ed04294b296bc6498ed1c7f121c2a9b63d0eecd8f75f3f0cca9e5447175a79b0"}
Jul 07 02:18:08.166110 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.166087    2570 generic.go:356] "Generic (PLEG): container finished" podID="2b740e4f-f613-4235-8aa1-ec671ed8208a" containerID="a2e692315d43cbe5184be2d1b2c3fb58be35c8130a91dc045ad033a243acbcfe" exitCode=0
Jul 07 02:18:08.166191 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.166150    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sk7b9" event={"ID":"2b740e4f-f613-4235-8aa1-ec671ed8208a","Type":"ContainerDied","Data":"a2e692315d43cbe5184be2d1b2c3fb58be35c8130a91dc045ad033a243acbcfe"}
Jul 07 02:18:08.167600 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.167579    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vsdng" event={"ID":"97b6993b-5c3c-400b-85fd-2044c0ae3eff","Type":"ContainerStarted","Data":"e79805edb1463f930f0f1427606bbff436ce6309fd08e1ace29c1a1a5ede355c"}
Jul 07 02:18:08.168980 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.168951    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr" event={"ID":"abebe566-c814-44bf-a8a0-c091d6d73fb4","Type":"ContainerStarted","Data":"f179a4c0e847260558a28f9f0cea84cdd6bdc366e7a9517e4c2fb79b51e3c0fe"}
Jul 07 02:18:08.170214 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.170197    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vdkb4" event={"ID":"549d5421-b20a-4288-88c3-c3629f566982","Type":"ContainerStarted","Data":"6bdff8b37ce498e42d1e43cc1f8c88dedefc58abee93314a1656d59d11ea0c4f"}
Jul 07 02:18:08.179982 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.179941    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h4v89" podStartSLOduration=4.970174839 podStartE2EDuration="24.17993091s" totalImagesPullingTime="19.209756071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:08.17970286 +0000 UTC m=+23.587292547" watchObservedRunningTime="2026-07-07 02:18:08.17993091 +0000 UTC m=+23.587520598"
Jul 07 02:18:08.180063 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.180041    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal" podStartSLOduration=23.180035224 podStartE2EDuration="23.180035224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:17:50.141011734 +0000 UTC m=+5.548601419" watchObservedRunningTime="2026-07-07 02:18:08.180035224 +0000 UTC m=+23.587624910"
Jul 07 02:18:08.250156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.250102    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vsdng" podStartSLOduration=4.020131652 podStartE2EDuration="23.250087031s" totalImagesPullingTime="19.229955379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:08.217895527 +0000 UTC m=+23.625485214" watchObservedRunningTime="2026-07-07 02:18:08.250087031 +0000 UTC m=+23.657676717"
Jul 07 02:18:08.273166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.273111    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x54gh" podStartSLOduration=4.062903427 podStartE2EDuration="23.273091903s" totalImagesPullingTime="19.210188476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:08.249475263 +0000 UTC m=+23.657064950" watchObservedRunningTime="2026-07-07 02:18:08.273091903 +0000 UTC m=+23.680681594"
Jul 07 02:18:08.327806 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.327760    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q27h5" podStartSLOduration=12.461748873 podStartE2EDuration="23.327745895s" totalImagesPullingTime="10.865997022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:08.275599823 +0000 UTC m=+23.683189502" watchObservedRunningTime="2026-07-07 02:18:08.327745895 +0000 UTC m=+23.735335641"
Jul 07 02:18:08.344209 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.344132    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vdkb4" podStartSLOduration=3.928595901 podStartE2EDuration="23.344120953s" totalImagesPullingTime="19.415525052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:08.343684807 +0000 UTC m=+23.751274506" watchObservedRunningTime="2026-07-07 02:18:08.344120953 +0000 UTC m=+23.751710641"
Jul 07 02:18:08.813778 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:08.813752    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 02:18:09.013395 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:09.013245    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T02:18:08.813774275Z","UUID":"d61b5c86-6031-49df-976e-e32018635f24","Handler":null,"Name":"","Endpoint":""}
Jul 07 02:18:09.014742 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:09.014721    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 02:18:09.014852 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:09.014755    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 02:18:09.081233 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:09.081205    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:09.081398 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:09.081358    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:09.081478 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:09.081401    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:09.081578 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:09.081556    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:18:09.173441 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:09.173410    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr" event={"ID":"abebe566-c814-44bf-a8a0-c091d6d73fb4","Type":"ContainerStarted","Data":"d5a662b2dcf0b78ef470464bc0b3d271482558f539c1fec37a3610591107586a"}
Jul 07 02:18:09.174705 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:09.174680    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7n4fq" event={"ID":"0b89780e-214b-4029-8ee8-386e60ec5437","Type":"ContainerStarted","Data":"40e9a3d5a5751a66f8e34e06cb634834f96e860e7edc9aa2a06e68e663313640"}
Jul 07 02:18:10.080918 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:10.080893    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:10.081115 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:10.081085    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:18:10.179633 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:10.179586    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr" event={"ID":"abebe566-c814-44bf-a8a0-c091d6d73fb4","Type":"ContainerStarted","Data":"bc987c785fed5b145666e040bab1bc1355a3e3cc37bfc81c73cfb2fa0e751358"}
Jul 07 02:18:10.187974 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:10.187937    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb" event={"ID":"68468352-f49f-497c-b807-a5e0d7e50082","Type":"ContainerStarted","Data":"c325781b83d88c3ff7261327313bcad53d510c90132993691821bbd6830b9906"}
Jul 07 02:18:10.197658 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:10.197620    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7n4fq" podStartSLOduration=6.984695758 podStartE2EDuration="26.197600153s" totalImagesPullingTime="19.212904395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:09.198023302 +0000 UTC m=+24.605612994" watchObservedRunningTime="2026-07-07 02:18:10.197600153 +0000 UTC m=+25.605189840"
Jul 07 02:18:10.197918 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:10.197896    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvpmr" podStartSLOduration=3.897050944 podStartE2EDuration="25.197890763s" totalImagesPullingTime="21.300839819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:10.197586307 +0000 UTC m=+25.605175994" watchObservedRunningTime="2026-07-07 02:18:10.197890763 +0000 UTC m=+25.605480472"
Jul 07 02:18:10.431235 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:10.431160    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:18:10.431702 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:10.431682    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:18:11.080916 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:11.080887    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:11.081073 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:11.080996    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:11.081073 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:11.081022    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:11.081073 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:11.081071    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:18:11.190064 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:11.190034    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:18:11.190586 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:11.190570    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q27h5"
Jul 07 02:18:12.081144 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:12.080956    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:12.081323 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:12.081235    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:18:13.081026 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:13.080991    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:13.081461 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:13.081009    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:13.081461 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:13.081137    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:13.081461 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:13.081194    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:18:14.081327 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:14.081163    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:14.081955 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:14.081411    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:18:14.201134 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:14.198477    2570 generic.go:356] "Generic (PLEG): container finished" podID="2b740e4f-f613-4235-8aa1-ec671ed8208a" containerID="140eed4c3c730176ddd71b56c870c42ec2566c79ac1908804f8d91971fad1d9f" exitCode=0
Jul 07 02:18:14.201134 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:14.198643    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sk7b9" event={"ID":"2b740e4f-f613-4235-8aa1-ec671ed8208a","Type":"ContainerDied","Data":"140eed4c3c730176ddd71b56c870c42ec2566c79ac1908804f8d91971fad1d9f"}
Jul 07 02:18:14.205052 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:14.205030    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb" event={"ID":"68468352-f49f-497c-b807-a5e0d7e50082","Type":"ContainerStarted","Data":"ef744826d1fb0c69c64ec2ec0de0bab736d031fe67e313030105e6da16793948"}
Jul 07 02:18:14.205330 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:14.205311    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:18:14.205436 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:14.205384    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:18:14.220652 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:14.220635    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:18:14.246336 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:14.246299    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb" podStartSLOduration=9.957098306 podStartE2EDuration="29.246286691s" totalImagesPullingTime="19.289188385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:14.245218203 +0000 UTC m=+29.652807912" watchObservedRunningTime="2026-07-07 02:18:14.246286691 +0000 UTC m=+29.653876378"
Jul 07 02:18:15.082800 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:15.082763    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:15.082800 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:15.082792    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:15.083448 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:15.082891    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:15.083448 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:15.082994    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:18:15.210574 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:15.210556    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:18:15.226560 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:15.226535    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:18:15.633888 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:15.633788    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pg4n4"]
Jul 07 02:18:15.634016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:15.633921    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:15.634071 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:15.634026    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:18:15.637914 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:15.637876    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7sr2n"]
Jul 07 02:18:15.638026 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:15.637985    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:15.638105 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:15.638083    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:18:15.638493 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:15.638471    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wcsrt"]
Jul 07 02:18:15.638595 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:15.638558    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:15.638653 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:15.638633    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:16.211241 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:16.211215    2570 generic.go:356] "Generic (PLEG): container finished" podID="2b740e4f-f613-4235-8aa1-ec671ed8208a" containerID="905e969076a6bc49cef0db85b649a0bbed8694a3191d2403ed78f7b3242da5f5" exitCode=0
Jul 07 02:18:16.211567 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:16.211288    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sk7b9" event={"ID":"2b740e4f-f613-4235-8aa1-ec671ed8208a","Type":"ContainerDied","Data":"905e969076a6bc49cef0db85b649a0bbed8694a3191d2403ed78f7b3242da5f5"}
Jul 07 02:18:17.081162 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:17.081125    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:17.081162 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:17.081153    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:17.081419 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:17.081156    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:17.081419 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:17.081252    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:18:17.081419 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:17.081317    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:18:17.081419 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:17.081409    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:17.215602 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:17.215527    2570 generic.go:356] "Generic (PLEG): container finished" podID="2b740e4f-f613-4235-8aa1-ec671ed8208a" containerID="a8725a6a0f698dacc7c84cd7191144d66cfb9af5aaea5b7d08d57557d33fa994" exitCode=0
Jul 07 02:18:17.215972 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:17.215598    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sk7b9" event={"ID":"2b740e4f-f613-4235-8aa1-ec671ed8208a","Type":"ContainerDied","Data":"a8725a6a0f698dacc7c84cd7191144d66cfb9af5aaea5b7d08d57557d33fa994"}
Jul 07 02:18:18.705517 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:18.705310    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:18.705956 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:18.705475    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:18.705956 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:18.705627    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs podName:e49d21df-0b01-4845-ad69-b2ad83b93187 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.705605297 +0000 UTC m=+66.113194963 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs") pod "network-metrics-daemon-wcsrt" (UID: "e49d21df-0b01-4845-ad69-b2ad83b93187") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:18.806521 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:18.806493    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:18.806694 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:18.806675    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:18.806756 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:18.806745    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret podName:818a88fe-5429-40b4-998f-cd34b2a149d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.80672758 +0000 UTC m=+66.214317258 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret") pod "global-pull-secret-syncer-pg4n4" (UID: "818a88fe-5429-40b4-998f-cd34b2a149d8") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:18.907022 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:18.906994    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdg6\" (UniqueName: \"kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6\") pod \"network-check-target-7sr2n\" (UID: \"c2b43d9f-097e-4e87-a0a6-40546d9005fe\") " pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:18.907166 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:18.907153    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:18.907228 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:18.907177    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:18.907228 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:18.907189    2570 projected.go:196] Error preparing data for projected volume kube-api-access-7hdg6 for pod openshift-network-diagnostics/network-check-target-7sr2n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:18.907328 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:18.907239    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6 podName:c2b43d9f-097e-4e87-a0a6-40546d9005fe nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.907223492 +0000 UTC m=+66.314813158 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7hdg6" (UniqueName: "kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6") pod "network-check-target-7sr2n" (UID: "c2b43d9f-097e-4e87-a0a6-40546d9005fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:19.081015 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.080986    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:19.081171 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.081028    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:19.081171 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:19.081105    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7sr2n" podUID="c2b43d9f-097e-4e87-a0a6-40546d9005fe"
Jul 07 02:18:19.081171 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.080986    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:19.081322 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:19.081202    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pg4n4" podUID="818a88fe-5429-40b4-998f-cd34b2a149d8"
Jul 07 02:18:19.081322 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:19.081291    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:18:19.954743 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.954719    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeReady"
Jul 07 02:18:19.955067 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.954831    2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 02:18:19.991532 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.991505    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"]
Jul 07 02:18:19.993258 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.993243    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:19.995673 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.995638    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 07 02:18:19.995787 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.995683    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ll4bl\""
Jul 07 02:18:19.995890 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.995874    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 07 02:18:19.995971 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.995951    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66bff4b87-w97mk"]
Jul 07 02:18:19.997553 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:19.997536    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.000913 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.000167    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 02:18:20.000913 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.000399    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 02:18:20.000913 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.000483    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 02:18:20.000913 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.000658    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kkr4v\""
Jul 07 02:18:20.005476 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.005432    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"]
Jul 07 02:18:20.006820 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.006799    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 02:18:20.011641 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.011625    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hfblg"]
Jul 07 02:18:20.013480 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.013467    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.016043 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.016026    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 02:18:20.016043 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.016035    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 02:18:20.016159 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.016055    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l8w8v\""
Jul 07 02:18:20.022182 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.022165    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rdp5n"]
Jul 07 02:18:20.027574 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.027552    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66bff4b87-w97mk"]
Jul 07 02:18:20.027670 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.027639    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rdp5n"
Jul 07 02:18:20.030523 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.030502    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 02:18:20.030645 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.030600    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 02:18:20.030717 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.030641    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pm56q\""
Jul 07 02:18:20.030972 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.030955    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 02:18:20.039627 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.039605    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rdp5n"]
Jul 07 02:18:20.040332 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.040316    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hfblg"]
Jul 07 02:18:20.115112 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115085    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:20.115312 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115119    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9982460-1009-4f83-8cfd-165fe6308074-cert\") pod \"ingress-canary-rdp5n\" (UID: \"d9982460-1009-4f83-8cfd-165fe6308074\") " pod="openshift-ingress-canary/ingress-canary-rdp5n"
Jul 07 02:18:20.115312 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115204    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34317344-6095-46da-b8ee-00d49db5a486-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:20.115312 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115242    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-bound-sa-token\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.115312 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115263    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-trusted-ca\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.115312 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115311    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.115548 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115329    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-installation-pull-secrets\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.115548 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115371    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qss22\" (UniqueName: \"kubernetes.io/projected/d9982460-1009-4f83-8cfd-165fe6308074-kube-api-access-qss22\") pod \"ingress-canary-rdp5n\" (UID: \"d9982460-1009-4f83-8cfd-165fe6308074\") " pod="openshift-ingress-canary/ingress-canary-rdp5n"
Jul 07 02:18:20.115548 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115416    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76gqn\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-kube-api-access-76gqn\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.115548 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115483    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-image-registry-private-configuration\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.115548 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115506    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0dd8bcac-f023-4c21-ad00-520136c355ac-ca-trust-extracted\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.115548 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115534    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17e64b2d-eebf-40dc-a342-909eb26792ba-config-volume\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.115815 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115560    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e64b2d-eebf-40dc-a342-909eb26792ba-tmp-dir\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.115815 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115629    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.115815 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115660    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-certificates\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.115815 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.115690    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cn5c\" (UniqueName: \"kubernetes.io/projected/17e64b2d-eebf-40dc-a342-909eb26792ba-kube-api-access-9cn5c\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.216382 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216340    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34317344-6095-46da-b8ee-00d49db5a486-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:20.216382 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216383    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-bound-sa-token\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.216568 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216400    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-trusted-ca\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.216568 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216420    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.216568 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216448    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-installation-pull-secrets\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.216568 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216470    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qss22\" (UniqueName: \"kubernetes.io/projected/d9982460-1009-4f83-8cfd-165fe6308074-kube-api-access-qss22\") pod \"ingress-canary-rdp5n\" (UID: \"d9982460-1009-4f83-8cfd-165fe6308074\") " pod="openshift-ingress-canary/ingress-canary-rdp5n"
Jul 07 02:18:20.216568 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216486    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76gqn\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-kube-api-access-76gqn\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.216568 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216504    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-image-registry-private-configuration\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.216568 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.216525    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:20.216945 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.216598    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls podName:17e64b2d-eebf-40dc-a342-909eb26792ba nodeName:}" failed. No retries permitted until 2026-07-07 02:18:20.716576943 +0000 UTC m=+36.124166608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls") pod "dns-default-hfblg" (UID: "17e64b2d-eebf-40dc-a342-909eb26792ba") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:20.216945 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216526    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0dd8bcac-f023-4c21-ad00-520136c355ac-ca-trust-extracted\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.216945 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216759    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17e64b2d-eebf-40dc-a342-909eb26792ba-config-volume\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.216945 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216788    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e64b2d-eebf-40dc-a342-909eb26792ba-tmp-dir\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.216945 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216829    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.216945 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216854    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-certificates\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.216945 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.216878    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cn5c\" (UniqueName: \"kubernetes.io/projected/17e64b2d-eebf-40dc-a342-909eb26792ba-kube-api-access-9cn5c\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.217337 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.216958    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 02:18:20.217337 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.216974    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bff4b87-w97mk: secret "image-registry-tls" not found
Jul 07 02:18:20.217337 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.217035    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls podName:0dd8bcac-f023-4c21-ad00-520136c355ac nodeName:}" failed. No retries permitted until 2026-07-07 02:18:20.717016261 +0000 UTC m=+36.124605954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls") pod "image-registry-66bff4b87-w97mk" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac") : secret "image-registry-tls" not found
Jul 07 02:18:20.217337 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.217180    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/34317344-6095-46da-b8ee-00d49db5a486-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:20.217337 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.217225    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:20.217337 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.217252    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9982460-1009-4f83-8cfd-165fe6308074-cert\") pod \"ingress-canary-rdp5n\" (UID: \"d9982460-1009-4f83-8cfd-165fe6308074\") " pod="openshift-ingress-canary/ingress-canary-rdp5n"
Jul 07 02:18:20.217337 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.217301    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17e64b2d-eebf-40dc-a342-909eb26792ba-config-volume\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.217337 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.217309    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e64b2d-eebf-40dc-a342-909eb26792ba-tmp-dir\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.217705 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.217408    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 07 02:18:20.217705 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.217441    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-trusted-ca\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.217705 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.217461    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert podName:34317344-6095-46da-b8ee-00d49db5a486 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:20.717446079 +0000 UTC m=+36.125035765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ljlsp" (UID: "34317344-6095-46da-b8ee-00d49db5a486") : secret "networking-console-plugin-cert" not found
Jul 07 02:18:20.217705 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.217472    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0dd8bcac-f023-4c21-ad00-520136c355ac-ca-trust-extracted\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.217705 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.217638    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-certificates\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.220812 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.220789    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-image-registry-private-configuration\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.220924 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.220831    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9982460-1009-4f83-8cfd-165fe6308074-cert\") pod \"ingress-canary-rdp5n\" (UID: \"d9982460-1009-4f83-8cfd-165fe6308074\") " pod="openshift-ingress-canary/ingress-canary-rdp5n"
Jul 07 02:18:20.220924 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.220870    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-installation-pull-secrets\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.228420 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.228388    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cn5c\" (UniqueName: \"kubernetes.io/projected/17e64b2d-eebf-40dc-a342-909eb26792ba-kube-api-access-9cn5c\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.228644 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.228625    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-bound-sa-token\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.228846 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.228828    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qss22\" (UniqueName: \"kubernetes.io/projected/d9982460-1009-4f83-8cfd-165fe6308074-kube-api-access-qss22\") pod \"ingress-canary-rdp5n\" (UID: \"d9982460-1009-4f83-8cfd-165fe6308074\") " pod="openshift-ingress-canary/ingress-canary-rdp5n"
Jul 07 02:18:20.228981 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.228962    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76gqn\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-kube-api-access-76gqn\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.338068 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.338037    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rdp5n"
Jul 07 02:18:20.488658 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.488615    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rdp5n"]
Jul 07 02:18:20.492997 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:18:20.492971    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9982460_1009_4f83_8cfd_165fe6308074.slice/crio-268fe9db3aa314324c9595db7e9bbed7863f7f4e75a9afbf84cf6ff8242ec505 WatchSource:0}: Error finding container 268fe9db3aa314324c9595db7e9bbed7863f7f4e75a9afbf84cf6ff8242ec505: Status 404 returned error can't find the container with id 268fe9db3aa314324c9595db7e9bbed7863f7f4e75a9afbf84cf6ff8242ec505
Jul 07 02:18:20.720752 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.720660    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:20.720901 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.720755    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:20.720901 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:20.720825    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:20.720901 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.720850    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 07 02:18:20.720901 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.720887    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:20.721111 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.720918    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert podName:34317344-6095-46da-b8ee-00d49db5a486 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:21.720902907 +0000 UTC m=+37.128492572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ljlsp" (UID: "34317344-6095-46da-b8ee-00d49db5a486") : secret "networking-console-plugin-cert" not found
Jul 07 02:18:20.721111 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.720920    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 02:18:20.721111 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.720937    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bff4b87-w97mk: secret "image-registry-tls" not found
Jul 07 02:18:20.721111 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.720959    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls podName:17e64b2d-eebf-40dc-a342-909eb26792ba nodeName:}" failed. No retries permitted until 2026-07-07 02:18:21.720941064 +0000 UTC m=+37.128530728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls") pod "dns-default-hfblg" (UID: "17e64b2d-eebf-40dc-a342-909eb26792ba") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:20.721111 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:20.720981    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls podName:0dd8bcac-f023-4c21-ad00-520136c355ac nodeName:}" failed. No retries permitted until 2026-07-07 02:18:21.720967258 +0000 UTC m=+37.128556939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls") pod "image-registry-66bff4b87-w97mk" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac") : secret "image-registry-tls" not found
Jul 07 02:18:21.081495 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.081455    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:21.081495 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.081473    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:21.082502 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.081757    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:21.084542 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.084519    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 02:18:21.084542 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.084534    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 02:18:21.085617 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.085589    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v4j68\""
Jul 07 02:18:21.085617 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.085614    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 02:18:21.085782 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.085589    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwgqf\""
Jul 07 02:18:21.085782 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.085602    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 02:18:21.226802 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.226767    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rdp5n" event={"ID":"d9982460-1009-4f83-8cfd-165fe6308074","Type":"ContainerStarted","Data":"268fe9db3aa314324c9595db7e9bbed7863f7f4e75a9afbf84cf6ff8242ec505"}
Jul 07 02:18:21.730770 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.730514    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:21.731004 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:21.730678    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 07 02:18:21.731004 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:21.730926    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert podName:34317344-6095-46da-b8ee-00d49db5a486 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:23.730905827 +0000 UTC m=+39.138495509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ljlsp" (UID: "34317344-6095-46da-b8ee-00d49db5a486") : secret "networking-console-plugin-cert" not found
Jul 07 02:18:21.731004 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.730844    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:21.731142 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:21.730947    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:21.731142 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:21.731031    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:21.731142 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:21.731085    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls podName:17e64b2d-eebf-40dc-a342-909eb26792ba nodeName:}" failed. No retries permitted until 2026-07-07 02:18:23.73106824 +0000 UTC m=+39.138657939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls") pod "dns-default-hfblg" (UID: "17e64b2d-eebf-40dc-a342-909eb26792ba") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:21.731258 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:21.731145    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 02:18:21.731258 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:21.731155    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bff4b87-w97mk: secret "image-registry-tls" not found
Jul 07 02:18:21.731258 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:21.731187    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls podName:0dd8bcac-f023-4c21-ad00-520136c355ac nodeName:}" failed. No retries permitted until 2026-07-07 02:18:23.731178636 +0000 UTC m=+39.138768300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls") pod "image-registry-66bff4b87-w97mk" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac") : secret "image-registry-tls" not found
Jul 07 02:18:22.232543 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:22.232517    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sk7b9" event={"ID":"2b740e4f-f613-4235-8aa1-ec671ed8208a","Type":"ContainerStarted","Data":"4ed833570f17e0e22117b9e1c83edb54fab33d73419eb07718e4c310bbccf148"}
Jul 07 02:18:23.238587 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:23.238545    2570 generic.go:356] "Generic (PLEG): container finished" podID="2b740e4f-f613-4235-8aa1-ec671ed8208a" containerID="4ed833570f17e0e22117b9e1c83edb54fab33d73419eb07718e4c310bbccf148" exitCode=0
Jul 07 02:18:23.239019 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:23.238618    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sk7b9" event={"ID":"2b740e4f-f613-4235-8aa1-ec671ed8208a","Type":"ContainerDied","Data":"4ed833570f17e0e22117b9e1c83edb54fab33d73419eb07718e4c310bbccf148"}
Jul 07 02:18:23.747000 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:23.746953    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:23.747172 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:23.747047    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:23.747172 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:23.747083    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:23.747249 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:23.747198    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 07 02:18:23.747282 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:23.747250    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert podName:34317344-6095-46da-b8ee-00d49db5a486 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:27.747234858 +0000 UTC m=+43.154824523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ljlsp" (UID: "34317344-6095-46da-b8ee-00d49db5a486") : secret "networking-console-plugin-cert" not found
Jul 07 02:18:23.747696 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:23.747675    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:23.747778 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:23.747733    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls podName:17e64b2d-eebf-40dc-a342-909eb26792ba nodeName:}" failed. No retries permitted until 2026-07-07 02:18:27.747720643 +0000 UTC m=+43.155310308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls") pod "dns-default-hfblg" (UID: "17e64b2d-eebf-40dc-a342-909eb26792ba") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:23.747837 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:23.747788    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 02:18:23.747837 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:23.747797    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bff4b87-w97mk: secret "image-registry-tls" not found
Jul 07 02:18:23.747837 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:23.747825    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls podName:0dd8bcac-f023-4c21-ad00-520136c355ac nodeName:}" failed. No retries permitted until 2026-07-07 02:18:27.747817512 +0000 UTC m=+43.155407181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls") pod "image-registry-66bff4b87-w97mk" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac") : secret "image-registry-tls" not found
Jul 07 02:18:24.244913 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:24.244883    2570 generic.go:356] "Generic (PLEG): container finished" podID="2b740e4f-f613-4235-8aa1-ec671ed8208a" containerID="8a939b4236aaafcd35082ecc1932b1e33a62162222b5e6a8033acdfcb1a4922c" exitCode=0
Jul 07 02:18:24.245271 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:24.244927    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sk7b9" event={"ID":"2b740e4f-f613-4235-8aa1-ec671ed8208a","Type":"ContainerDied","Data":"8a939b4236aaafcd35082ecc1932b1e33a62162222b5e6a8033acdfcb1a4922c"}
Jul 07 02:18:25.249723 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:25.249688    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sk7b9" event={"ID":"2b740e4f-f613-4235-8aa1-ec671ed8208a","Type":"ContainerStarted","Data":"5a0b5f75964856905dba8368a50b9446430a6e0a2e1cb944e46fe546e72527ff"}
Jul 07 02:18:25.251064 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:25.251036    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rdp5n" event={"ID":"d9982460-1009-4f83-8cfd-165fe6308074","Type":"ContainerStarted","Data":"ba9564344a79941b98df6c0f99c84eff58b779a8c817eb4799a7001c9a72cb9f"}
Jul 07 02:18:25.272932 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:25.272852    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sk7b9" podStartSLOduration=8.936931011 podStartE2EDuration="40.272840194s" totalImagesPullingTime="31.335909183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:25.270796321 +0000 UTC m=+40.678386008" watchObservedRunningTime="2026-07-07 02:18:25.272840194 +0000 UTC m=+40.680429880"
Jul 07 02:18:25.287950 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:25.287910    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rdp5n" podStartSLOduration=2.6193762019999998 podStartE2EDuration="6.287897122s" totalImagesPullingTime="3.66852092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:25.287225862 +0000 UTC m=+40.694815549" watchObservedRunningTime="2026-07-07 02:18:25.287897122 +0000 UTC m=+40.695486836"
Jul 07 02:18:27.776081 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:27.776042    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:27.776534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:27.776095    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:27.776534 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:27.776198    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 02:18:27.776534 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:27.776208    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bff4b87-w97mk: secret "image-registry-tls" not found
Jul 07 02:18:27.776534 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:27.776217    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:27.776534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:27.776224    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:27.776534 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:27.776250    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls podName:0dd8bcac-f023-4c21-ad00-520136c355ac nodeName:}" failed. No retries permitted until 2026-07-07 02:18:35.776238008 +0000 UTC m=+51.183827675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls") pod "image-registry-66bff4b87-w97mk" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac") : secret "image-registry-tls" not found
Jul 07 02:18:27.776534 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:27.776292    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 07 02:18:27.776534 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:27.776294    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls podName:17e64b2d-eebf-40dc-a342-909eb26792ba nodeName:}" failed. No retries permitted until 2026-07-07 02:18:35.776275425 +0000 UTC m=+51.183865104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls") pod "dns-default-hfblg" (UID: "17e64b2d-eebf-40dc-a342-909eb26792ba") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:27.776534 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:27.776334    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert podName:34317344-6095-46da-b8ee-00d49db5a486 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:35.776322036 +0000 UTC m=+51.183911733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ljlsp" (UID: "34317344-6095-46da-b8ee-00d49db5a486") : secret "networking-console-plugin-cert" not found
Jul 07 02:18:35.830133 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:35.830099    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:35.830581 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:35.830143    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:35.830581 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:35.830191    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:35.830581 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:35.830243    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 02:18:35.830581 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:35.830261    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bff4b87-w97mk: secret "image-registry-tls" not found
Jul 07 02:18:35.830581 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:35.830326    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls podName:0dd8bcac-f023-4c21-ad00-520136c355ac nodeName:}" failed. No retries permitted until 2026-07-07 02:18:51.83030378 +0000 UTC m=+67.237893445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls") pod "image-registry-66bff4b87-w97mk" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac") : secret "image-registry-tls" not found
Jul 07 02:18:35.830581 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:35.830387    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:35.830581 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:35.830387    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 07 02:18:35.830581 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:35.830456    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls podName:17e64b2d-eebf-40dc-a342-909eb26792ba nodeName:}" failed. No retries permitted until 2026-07-07 02:18:51.830432229 +0000 UTC m=+67.238021893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls") pod "dns-default-hfblg" (UID: "17e64b2d-eebf-40dc-a342-909eb26792ba") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:35.830581 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:35.830512    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert podName:34317344-6095-46da-b8ee-00d49db5a486 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:51.830496583 +0000 UTC m=+67.238086248 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ljlsp" (UID: "34317344-6095-46da-b8ee-00d49db5a486") : secret "networking-console-plugin-cert" not found
Jul 07 02:18:47.226832 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:47.226800    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcklb"
Jul 07 02:18:50.731852 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:50.731816    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:18:50.738077 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:50.738059    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 02:18:50.742155 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:50.742142    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 02:18:50.742214 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:50.742192    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs podName:e49d21df-0b01-4845-ad69-b2ad83b93187 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:54.742177106 +0000 UTC m=+130.149766771 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs") pod "network-metrics-daemon-wcsrt" (UID: "e49d21df-0b01-4845-ad69-b2ad83b93187") : secret "metrics-daemon-secret" not found
Jul 07 02:18:50.832181 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:50.832156    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:50.834609 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:50.834591    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 02:18:50.845682 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:50.845653    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a88fe-5429-40b4-998f-cd34b2a149d8-original-pull-secret\") pod \"global-pull-secret-syncer-pg4n4\" (UID: \"818a88fe-5429-40b4-998f-cd34b2a149d8\") " pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:50.933188 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:50.933162    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdg6\" (UniqueName: \"kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6\") pod \"network-check-target-7sr2n\" (UID: \"c2b43d9f-097e-4e87-a0a6-40546d9005fe\") " pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:50.936137 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:50.936121    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 02:18:50.946053 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:50.946032    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 02:18:50.957174 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:50.957152    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdg6\" (UniqueName: \"kubernetes.io/projected/c2b43d9f-097e-4e87-a0a6-40546d9005fe-kube-api-access-7hdg6\") pod \"network-check-target-7sr2n\" (UID: \"c2b43d9f-097e-4e87-a0a6-40546d9005fe\") " pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:51.093486 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:51.093461    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pg4n4"
Jul 07 02:18:51.107672 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:51.107649    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v4j68\""
Jul 07 02:18:51.115581 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:51.115558    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:51.245235 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:51.245208    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pg4n4"]
Jul 07 02:18:51.248190 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:18:51.248162    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818a88fe_5429_40b4_998f_cd34b2a149d8.slice/crio-00822f8cd22a3b5b57eb257351a96f00799b8a95abdefcadcce99fcbe99446a4 WatchSource:0}: Error finding container 00822f8cd22a3b5b57eb257351a96f00799b8a95abdefcadcce99fcbe99446a4: Status 404 returned error can't find the container with id 00822f8cd22a3b5b57eb257351a96f00799b8a95abdefcadcce99fcbe99446a4
Jul 07 02:18:51.262026 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:51.262004    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7sr2n"]
Jul 07 02:18:51.264696 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:18:51.264673    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b43d9f_097e_4e87_a0a6_40546d9005fe.slice/crio-fa4281b5ea1bb37a5b034cbe8ab0f0b1074cbae389b80c5ab0df6c9976aa8416 WatchSource:0}: Error finding container fa4281b5ea1bb37a5b034cbe8ab0f0b1074cbae389b80c5ab0df6c9976aa8416: Status 404 returned error can't find the container with id fa4281b5ea1bb37a5b034cbe8ab0f0b1074cbae389b80c5ab0df6c9976aa8416
Jul 07 02:18:51.310880 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:51.310851    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7sr2n" event={"ID":"c2b43d9f-097e-4e87-a0a6-40546d9005fe","Type":"ContainerStarted","Data":"fa4281b5ea1bb37a5b034cbe8ab0f0b1074cbae389b80c5ab0df6c9976aa8416"}
Jul 07 02:18:51.311993 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:51.311970    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pg4n4" event={"ID":"818a88fe-5429-40b4-998f-cd34b2a149d8","Type":"ContainerStarted","Data":"00822f8cd22a3b5b57eb257351a96f00799b8a95abdefcadcce99fcbe99446a4"}
Jul 07 02:18:51.840981 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:51.840944    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:18:51.841415 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:51.840998    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:18:51.841415 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:51.841044    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:18:51.841415 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:51.841094    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 02:18:51.841415 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:51.841111    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bff4b87-w97mk: secret "image-registry-tls" not found
Jul 07 02:18:51.841415 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:51.841128    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:51.841415 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:51.841165    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls podName:0dd8bcac-f023-4c21-ad00-520136c355ac nodeName:}" failed. No retries permitted until 2026-07-07 02:19:23.841147134 +0000 UTC m=+99.248736804 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls") pod "image-registry-66bff4b87-w97mk" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac") : secret "image-registry-tls" not found
Jul 07 02:18:51.841415 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:51.841180    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls podName:17e64b2d-eebf-40dc-a342-909eb26792ba nodeName:}" failed. No retries permitted until 2026-07-07 02:19:23.841173756 +0000 UTC m=+99.248763421 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls") pod "dns-default-hfblg" (UID: "17e64b2d-eebf-40dc-a342-909eb26792ba") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:51.841415 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:51.841201    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 07 02:18:51.841415 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:18:51.841228    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert podName:34317344-6095-46da-b8ee-00d49db5a486 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:23.841217677 +0000 UTC m=+99.248807341 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ljlsp" (UID: "34317344-6095-46da-b8ee-00d49db5a486") : secret "networking-console-plugin-cert" not found
Jul 07 02:18:56.327190 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:56.327152    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7sr2n" event={"ID":"c2b43d9f-097e-4e87-a0a6-40546d9005fe","Type":"ContainerStarted","Data":"932ef47cf803d68ea85f5a1220c574fc49083e73d792f50618a04b7f4e93052c"}
Jul 07 02:18:56.327611 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:56.327278    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:18:56.328745 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:56.328721    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pg4n4" event={"ID":"818a88fe-5429-40b4-998f-cd34b2a149d8","Type":"ContainerStarted","Data":"49c44fb725fa33959b35bf690534ecc953c6f33ae96c7ad25232db93dfd5b8e6"}
Jul 07 02:18:56.346065 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:56.345999    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7sr2n" podStartSLOduration=66.907957261 podStartE2EDuration="1m11.345986977s" totalImagesPullingTime="4.438029716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:56.345791215 +0000 UTC m=+71.753380902" watchObservedRunningTime="2026-07-07 02:18:56.345986977 +0000 UTC m=+71.753576658"
Jul 07 02:18:56.359415 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:18:56.359369    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pg4n4" podStartSLOduration=66.90218912 podStartE2EDuration="1m11.359337413s" totalImagesPullingTime="4.457148293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:56.358472196 +0000 UTC m=+71.766061885" watchObservedRunningTime="2026-07-07 02:18:56.359337413 +0000 UTC m=+71.766927100"
Jul 07 02:19:23.856464 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:19:23.856418    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:19:23.857053 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:19:23.856480    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:19:23.857053 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:19:23.856507    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:19:23.857053 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:19:23.856578    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:19:23.857053 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:19:23.856615    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 07 02:19:23.857053 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:19:23.856657    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls podName:17e64b2d-eebf-40dc-a342-909eb26792ba nodeName:}" failed. No retries permitted until 2026-07-07 02:20:27.856638247 +0000 UTC m=+163.264227926 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls") pod "dns-default-hfblg" (UID: "17e64b2d-eebf-40dc-a342-909eb26792ba") : secret "dns-default-metrics-tls" not found
Jul 07 02:19:23.857053 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:19:23.856667    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 02:19:23.857053 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:19:23.856688    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bff4b87-w97mk: secret "image-registry-tls" not found
Jul 07 02:19:23.857053 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:19:23.856674    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert podName:34317344-6095-46da-b8ee-00d49db5a486 nodeName:}" failed. No retries permitted until 2026-07-07 02:20:27.856667577 +0000 UTC m=+163.264257242 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ljlsp" (UID: "34317344-6095-46da-b8ee-00d49db5a486") : secret "networking-console-plugin-cert" not found
Jul 07 02:19:23.857053 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:19:23.856790    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls podName:0dd8bcac-f023-4c21-ad00-520136c355ac nodeName:}" failed. No retries permitted until 2026-07-07 02:20:27.856770271 +0000 UTC m=+163.264359945 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls") pod "image-registry-66bff4b87-w97mk" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac") : secret "image-registry-tls" not found
Jul 07 02:19:27.333157 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:19:27.333125    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7sr2n"
Jul 07 02:19:54.761120 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:19:54.761072    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:19:54.761616 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:19:54.761221    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 02:19:54.761616 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:19:54.761288    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs podName:e49d21df-0b01-4845-ad69-b2ad83b93187 nodeName:}" failed. No retries permitted until 2026-07-07 02:21:56.76127217 +0000 UTC m=+252.168861834 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs") pod "network-metrics-daemon-wcsrt" (UID: "e49d21df-0b01-4845-ad69-b2ad83b93187") : secret "metrics-daemon-secret" not found
Jul 07 02:20:23.003469 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:20:23.003416    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp" podUID="34317344-6095-46da-b8ee-00d49db5a486"
Jul 07 02:20:23.010592 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:20:23.010556    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-66bff4b87-w97mk" podUID="0dd8bcac-f023-4c21-ad00-520136c355ac"
Jul 07 02:20:23.022731 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:20:23.022704    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hfblg" podUID="17e64b2d-eebf-40dc-a342-909eb26792ba"
Jul 07 02:20:23.526583 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:20:23.526551    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hfblg"
Jul 07 02:20:23.526583 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:20:23.526565    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:20:23.526805 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:20:23.526555    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:20:24.099856 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:20:24.099823    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:20:27.880526 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:20:27.880490    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:20:27.880863 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:20:27.880540    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:20:27.880863 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:20:27.880624    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 02:20:27.880863 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:20:27.880634    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66bff4b87-w97mk: secret "image-registry-tls" not found
Jul 07 02:20:27.880863 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:20:27.880636    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:20:27.880863 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:20:27.880657    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:20:27.880863 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:20:27.880686    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls podName:0dd8bcac-f023-4c21-ad00-520136c355ac nodeName:}" failed. No retries permitted until 2026-07-07 02:22:29.880673424 +0000 UTC m=+285.288263089 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls") pod "image-registry-66bff4b87-w97mk" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac") : secret "image-registry-tls" not found
Jul 07 02:20:27.880863 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:20:27.880731    2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 07 02:20:27.880863 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:20:27.880739    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls podName:17e64b2d-eebf-40dc-a342-909eb26792ba nodeName:}" failed. No retries permitted until 2026-07-07 02:22:29.880716572 +0000 UTC m=+285.288306245 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls") pod "dns-default-hfblg" (UID: "17e64b2d-eebf-40dc-a342-909eb26792ba") : secret "dns-default-metrics-tls" not found
Jul 07 02:20:27.880863 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:20:27.880783    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert podName:34317344-6095-46da-b8ee-00d49db5a486 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:29.880771118 +0000 UTC m=+285.288360796 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-ljlsp" (UID: "34317344-6095-46da-b8ee-00d49db5a486") : secret "networking-console-plugin-cert" not found
Jul 07 02:20:37.081504 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:20:37.081421    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:21:56.770791 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:21:56.770748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:21:56.771328 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:21:56.770896    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 02:21:56.771328 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:21:56.770973    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs podName:e49d21df-0b01-4845-ad69-b2ad83b93187 nodeName:}" failed. No retries permitted until 2026-07-07 02:23:58.770955867 +0000 UTC m=+374.178545532 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs") pod "network-metrics-daemon-wcsrt" (UID: "e49d21df-0b01-4845-ad69-b2ad83b93187") : secret "metrics-daemon-secret" not found
Jul 07 02:22:05.394012 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.393983    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cpjkf"]
Jul 07 02:22:05.395680 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.395665    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.399501 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.399440    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 07 02:22:05.399687 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.399440    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 07 02:22:05.399765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.399732    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j4s55\""
Jul 07 02:22:05.399892 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.399866    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"]
Jul 07 02:22:05.400423 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.400402    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 02:22:05.400534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.400443    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 02:22:05.401905 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.401887    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"]
Jul 07 02:22:05.402054 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.402035    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:05.403687 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.403669    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"]
Jul 07 02:22:05.403781 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.403768    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"
Jul 07 02:22:05.403832 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.403805    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 07 02:22:05.404541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.404521    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 02:22:05.405414 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.405396    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"
Jul 07 02:22:05.406995 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.406972    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 02:22:05.407244 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.407210    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 07 02:22:05.407537 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.407516    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 07 02:22:05.407733 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.407551    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 07 02:22:05.407808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.407601    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 07 02:22:05.408093 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.408073    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 07 02:22:05.409316 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.409296    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4pj9k\""
Jul 07 02:22:05.412516 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.412493    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:22:05.412671 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.412496    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wsv7s\""
Jul 07 02:22:05.412671 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.412496    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5m6pz\""
Jul 07 02:22:05.412825 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.412540    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 07 02:22:05.412825 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.412542    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 07 02:22:05.412825 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.412599    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:22:05.412825 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.412611    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 07 02:22:05.413654 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.413627    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cpjkf"]
Jul 07 02:22:05.422978 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.422961    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"]
Jul 07 02:22:05.423946 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.423923    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"]
Jul 07 02:22:05.429463 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.429445    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"]
Jul 07 02:22:05.485242 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.485218    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-jlf4s"]
Jul 07 02:22:05.487494 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.487480    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-jlf4s"
Jul 07 02:22:05.489630 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.489611    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:22:05.489630 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.489624    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dkl25\""
Jul 07 02:22:05.489804 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.489626    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 07 02:22:05.491750 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.491732    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"]
Jul 07 02:22:05.493819 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.493802    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-996f5cb75-q2sjq"]
Jul 07 02:22:05.494084 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.494064    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:05.495732 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.495713    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.496564 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.496545    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zflbh\""
Jul 07 02:22:05.496747 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.496726    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 07 02:22:05.497051 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.497036    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:22:05.497136 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.497114    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 07 02:22:05.498546 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.498530    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 07 02:22:05.498789 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.498772    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 07 02:22:05.498917 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.498901    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 07 02:22:05.499109 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.499087    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rkcjj\""
Jul 07 02:22:05.499226 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.499206    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 02:22:05.499325 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.499292    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 02:22:05.499500 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.498785    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 07 02:22:05.499918 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.499901    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-jlf4s"]
Jul 07 02:22:05.505524 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.505505    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"]
Jul 07 02:22:05.514172 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.514152    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-996f5cb75-q2sjq"]
Jul 07 02:22:05.528496 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528472    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f82c493c-8343-46b9-a50c-5dc1414d66f0-config\") pod \"service-ca-operator-5d46d9ffd4-gckv4\" (UID: \"f82c493c-8343-46b9-a50c-5dc1414d66f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"
Jul 07 02:22:05.528593 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528506    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f82c493c-8343-46b9-a50c-5dc1414d66f0-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-gckv4\" (UID: \"f82c493c-8343-46b9-a50c-5dc1414d66f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"
Jul 07 02:22:05.528593 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528532    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r462p\" (UniqueName: \"kubernetes.io/projected/f82c493c-8343-46b9-a50c-5dc1414d66f0-kube-api-access-r462p\") pod \"service-ca-operator-5d46d9ffd4-gckv4\" (UID: \"f82c493c-8343-46b9-a50c-5dc1414d66f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"
Jul 07 02:22:05.528593 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528559    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b3022fc-1859-4338-bb06-cef488979501-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-jz256\" (UID: \"6b3022fc-1859-4338-bb06-cef488979501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"
Jul 07 02:22:05.528776 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528615    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f681b31-f05e-4f17-8b0d-7aaac08ea039-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.528776 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528640    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b3022fc-1859-4338-bb06-cef488979501-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-jz256\" (UID: \"6b3022fc-1859-4338-bb06-cef488979501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"
Jul 07 02:22:05.528776 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528667    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f681b31-f05e-4f17-8b0d-7aaac08ea039-snapshots\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.528776 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528694    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmf6\" (UniqueName: \"kubernetes.io/projected/6b3022fc-1859-4338-bb06-cef488979501-kube-api-access-rkmf6\") pod \"kube-storage-version-migrator-operator-846bf668f7-jz256\" (UID: \"6b3022fc-1859-4338-bb06-cef488979501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"
Jul 07 02:22:05.528776 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528726    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f681b31-f05e-4f17-8b0d-7aaac08ea039-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.528776 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528751    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f681b31-f05e-4f17-8b0d-7aaac08ea039-serving-cert\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.529101 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528777    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:05.529101 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528807    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj8cv\" (UniqueName: \"kubernetes.io/projected/9f681b31-f05e-4f17-8b0d-7aaac08ea039-kube-api-access-jj8cv\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.529101 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528832    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ts2\" (UniqueName: \"kubernetes.io/projected/a15d589e-a296-4ba9-83d9-2a8a74f8823e-kube-api-access-m8ts2\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:05.529101 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528859    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a15d589e-a296-4ba9-83d9-2a8a74f8823e-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:05.529101 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.528979    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f681b31-f05e-4f17-8b0d-7aaac08ea039-tmp\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.629954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.629926    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f82c493c-8343-46b9-a50c-5dc1414d66f0-config\") pod \"service-ca-operator-5d46d9ffd4-gckv4\" (UID: \"f82c493c-8343-46b9-a50c-5dc1414d66f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"
Jul 07 02:22:05.629954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.629955    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f82c493c-8343-46b9-a50c-5dc1414d66f0-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-gckv4\" (UID: \"f82c493c-8343-46b9-a50c-5dc1414d66f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"
Jul 07 02:22:05.630096 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.629974    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r462p\" (UniqueName: \"kubernetes.io/projected/f82c493c-8343-46b9-a50c-5dc1414d66f0-kube-api-access-r462p\") pod \"service-ca-operator-5d46d9ffd4-gckv4\" (UID: \"f82c493c-8343-46b9-a50c-5dc1414d66f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"
Jul 07 02:22:05.630096 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.629990    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b3022fc-1859-4338-bb06-cef488979501-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-jz256\" (UID: \"6b3022fc-1859-4338-bb06-cef488979501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"
Jul 07 02:22:05.630096 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630010    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f681b31-f05e-4f17-8b0d-7aaac08ea039-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.630096 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630028    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b3022fc-1859-4338-bb06-cef488979501-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-jz256\" (UID: \"6b3022fc-1859-4338-bb06-cef488979501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"
Jul 07 02:22:05.630096 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630048    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f681b31-f05e-4f17-8b0d-7aaac08ea039-snapshots\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.630096 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630064    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmf6\" (UniqueName: \"kubernetes.io/projected/6b3022fc-1859-4338-bb06-cef488979501-kube-api-access-rkmf6\") pod \"kube-storage-version-migrator-operator-846bf668f7-jz256\" (UID: \"6b3022fc-1859-4338-bb06-cef488979501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"
Jul 07 02:22:05.630096 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630084    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f681b31-f05e-4f17-8b0d-7aaac08ea039-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.630450 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630104    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vjxfv\" (UID: \"40be13c4-b1af-4941-8d7b-3ad560fefb56\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:05.630450 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630129    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f681b31-f05e-4f17-8b0d-7aaac08ea039-serving-cert\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.630450 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630239    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:05.630450 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630299    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj8cv\" (UniqueName: \"kubernetes.io/projected/9f681b31-f05e-4f17-8b0d-7aaac08ea039-kube-api-access-jj8cv\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.630450 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630327    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ts2\" (UniqueName: \"kubernetes.io/projected/a15d589e-a296-4ba9-83d9-2a8a74f8823e-kube-api-access-m8ts2\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:05.630450 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630373    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.630450 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630414    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a15d589e-a296-4ba9-83d9-2a8a74f8823e-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:05.630450 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630440    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894gh\" (UniqueName: \"kubernetes.io/projected/40be13c4-b1af-4941-8d7b-3ad560fefb56-kube-api-access-894gh\") pod \"cluster-samples-operator-56fcf86bf-vjxfv\" (UID: \"40be13c4-b1af-4941-8d7b-3ad560fefb56\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:05.630833 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630498    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f681b31-f05e-4f17-8b0d-7aaac08ea039-tmp\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.630833 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630526    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-default-certificate\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.630833 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630548    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-stats-auth\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.630833 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630575    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc5dh\" (UniqueName: \"kubernetes.io/projected/a482dca7-e069-4636-aefe-bf75e2d4ed17-kube-api-access-nc5dh\") pod \"volume-data-source-validator-5d456fdbcd-jlf4s\" (UID: \"a482dca7-e069-4636-aefe-bf75e2d4ed17\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-jlf4s"
Jul 07 02:22:05.630833 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630616    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4zh7\" (UniqueName: \"kubernetes.io/projected/e897bc4a-e67a-4a06-bb96-64b781e167c7-kube-api-access-f4zh7\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.630833 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:05.630663    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:05.630833 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630693    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b3022fc-1859-4338-bb06-cef488979501-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-jz256\" (UID: \"6b3022fc-1859-4338-bb06-cef488979501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"
Jul 07 02:22:05.630833 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630663    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.630833 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:05.630732    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls podName:a15d589e-a296-4ba9-83d9-2a8a74f8823e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:06.130713916 +0000 UTC m=+261.538303582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-t678z" (UID: "a15d589e-a296-4ba9-83d9-2a8a74f8823e") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:05.630833 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630661    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f681b31-f05e-4f17-8b0d-7aaac08ea039-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.630833 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.630789    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f681b31-f05e-4f17-8b0d-7aaac08ea039-snapshots\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.631391 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.631012    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f681b31-f05e-4f17-8b0d-7aaac08ea039-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.631391 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.631087    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f681b31-f05e-4f17-8b0d-7aaac08ea039-tmp\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.631542 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.631519    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f82c493c-8343-46b9-a50c-5dc1414d66f0-config\") pod \"service-ca-operator-5d46d9ffd4-gckv4\" (UID: \"f82c493c-8343-46b9-a50c-5dc1414d66f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"
Jul 07 02:22:05.631802 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.631593    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a15d589e-a296-4ba9-83d9-2a8a74f8823e-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:05.632639 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.632618    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b3022fc-1859-4338-bb06-cef488979501-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-jz256\" (UID: \"6b3022fc-1859-4338-bb06-cef488979501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"
Jul 07 02:22:05.632973 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.632953    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f681b31-f05e-4f17-8b0d-7aaac08ea039-serving-cert\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.633125 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.633109    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f82c493c-8343-46b9-a50c-5dc1414d66f0-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-gckv4\" (UID: \"f82c493c-8343-46b9-a50c-5dc1414d66f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"
Jul 07 02:22:05.639648 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.639619    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmf6\" (UniqueName: \"kubernetes.io/projected/6b3022fc-1859-4338-bb06-cef488979501-kube-api-access-rkmf6\") pod \"kube-storage-version-migrator-operator-846bf668f7-jz256\" (UID: \"6b3022fc-1859-4338-bb06-cef488979501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"
Jul 07 02:22:05.639766 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.639737    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r462p\" (UniqueName: \"kubernetes.io/projected/f82c493c-8343-46b9-a50c-5dc1414d66f0-kube-api-access-r462p\") pod \"service-ca-operator-5d46d9ffd4-gckv4\" (UID: \"f82c493c-8343-46b9-a50c-5dc1414d66f0\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"
Jul 07 02:22:05.639867 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.639849    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj8cv\" (UniqueName: \"kubernetes.io/projected/9f681b31-f05e-4f17-8b0d-7aaac08ea039-kube-api-access-jj8cv\") pod \"insights-operator-5c847dcfc4-cpjkf\" (UID: \"9f681b31-f05e-4f17-8b0d-7aaac08ea039\") " pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.640071 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.640047    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ts2\" (UniqueName: \"kubernetes.io/projected/a15d589e-a296-4ba9-83d9-2a8a74f8823e-kube-api-access-m8ts2\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:05.707463 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.707406    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf"
Jul 07 02:22:05.723122 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.723103    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"
Jul 07 02:22:05.727818 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.727801    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"
Jul 07 02:22:05.731897 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.731725    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vjxfv\" (UID: \"40be13c4-b1af-4941-8d7b-3ad560fefb56\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:05.731897 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.731791    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.731897 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.731823    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894gh\" (UniqueName: \"kubernetes.io/projected/40be13c4-b1af-4941-8d7b-3ad560fefb56-kube-api-access-894gh\") pod \"cluster-samples-operator-56fcf86bf-vjxfv\" (UID: \"40be13c4-b1af-4941-8d7b-3ad560fefb56\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:05.732285 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.731875    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-default-certificate\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.732285 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.732152    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-stats-auth\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.732285 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.732191    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc5dh\" (UniqueName: \"kubernetes.io/projected/a482dca7-e069-4636-aefe-bf75e2d4ed17-kube-api-access-nc5dh\") pod \"volume-data-source-validator-5d456fdbcd-jlf4s\" (UID: \"a482dca7-e069-4636-aefe-bf75e2d4ed17\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-jlf4s"
Jul 07 02:22:05.732285 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.732239    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4zh7\" (UniqueName: \"kubernetes.io/projected/e897bc4a-e67a-4a06-bb96-64b781e167c7-kube-api-access-f4zh7\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.732912 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:05.732719    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 07 02:22:05.732912 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:05.732777    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls podName:40be13c4-b1af-4941-8d7b-3ad560fefb56 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:06.232757573 +0000 UTC m=+261.640347237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vjxfv" (UID: "40be13c4-b1af-4941-8d7b-3ad560fefb56") : secret "samples-operator-tls" not found
Jul 07 02:22:05.732912 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:05.732802    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 02:22:05.732912 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:05.732869    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs podName:e897bc4a-e67a-4a06-bb96-64b781e167c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:06.232848688 +0000 UTC m=+261.640438353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs") pod "router-default-996f5cb75-q2sjq" (UID: "e897bc4a-e67a-4a06-bb96-64b781e167c7") : secret "router-metrics-certs-default" not found
Jul 07 02:22:05.733145 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.732983    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.733145 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:05.733135    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle podName:e897bc4a-e67a-4a06-bb96-64b781e167c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:06.233119312 +0000 UTC m=+261.640708996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle") pod "router-default-996f5cb75-q2sjq" (UID: "e897bc4a-e67a-4a06-bb96-64b781e167c7") : configmap references non-existent config key: service-ca.crt
Jul 07 02:22:05.735738 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.735714    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-stats-auth\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.735834 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.735717    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-default-certificate\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.741275 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.741243    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc5dh\" (UniqueName: \"kubernetes.io/projected/a482dca7-e069-4636-aefe-bf75e2d4ed17-kube-api-access-nc5dh\") pod \"volume-data-source-validator-5d456fdbcd-jlf4s\" (UID: \"a482dca7-e069-4636-aefe-bf75e2d4ed17\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-jlf4s"
Jul 07 02:22:05.741451 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.741432    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4zh7\" (UniqueName: \"kubernetes.io/projected/e897bc4a-e67a-4a06-bb96-64b781e167c7-kube-api-access-f4zh7\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:05.741811 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.741789    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-894gh\" (UniqueName: \"kubernetes.io/projected/40be13c4-b1af-4941-8d7b-3ad560fefb56-kube-api-access-894gh\") pod \"cluster-samples-operator-56fcf86bf-vjxfv\" (UID: \"40be13c4-b1af-4941-8d7b-3ad560fefb56\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:05.798979 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.798912    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-jlf4s"
Jul 07 02:22:05.853410 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.853277    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cpjkf"]
Jul 07 02:22:05.871624 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.871604    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256"]
Jul 07 02:22:05.893576 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.893550    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4"]
Jul 07 02:22:05.896476 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:22:05.896450    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82c493c_8343_46b9_a50c_5dc1414d66f0.slice/crio-38a961de00f6e88c2a1091f5ba2b593a05085a21bfa9cbfe0a758b5bd9c07366 WatchSource:0}: Error finding container 38a961de00f6e88c2a1091f5ba2b593a05085a21bfa9cbfe0a758b5bd9c07366: Status 404 returned error can't find the container with id 38a961de00f6e88c2a1091f5ba2b593a05085a21bfa9cbfe0a758b5bd9c07366
Jul 07 02:22:05.938481 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:05.938455    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-jlf4s"]
Jul 07 02:22:05.941842 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:22:05.941819    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda482dca7_e069_4636_aefe_bf75e2d4ed17.slice/crio-13e6d989e32b0d9f393c0547faf837eec7eb685072cdc3d6986051e5dab441cd WatchSource:0}: Error finding container 13e6d989e32b0d9f393c0547faf837eec7eb685072cdc3d6986051e5dab441cd: Status 404 returned error can't find the container with id 13e6d989e32b0d9f393c0547faf837eec7eb685072cdc3d6986051e5dab441cd
Jul 07 02:22:06.136386 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:06.136341    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:06.136500 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:06.136486    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:06.136552 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:06.136541    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls podName:a15d589e-a296-4ba9-83d9-2a8a74f8823e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:07.136527213 +0000 UTC m=+262.544116878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-t678z" (UID: "a15d589e-a296-4ba9-83d9-2a8a74f8823e") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:06.236926 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:06.236881    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:06.237016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:06.236929    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vjxfv\" (UID: \"40be13c4-b1af-4941-8d7b-3ad560fefb56\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:06.237016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:06.236956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:06.237126 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:06.237030    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 02:22:06.237126 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:06.237044    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle podName:e897bc4a-e67a-4a06-bb96-64b781e167c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:07.237025882 +0000 UTC m=+262.644615549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle") pod "router-default-996f5cb75-q2sjq" (UID: "e897bc4a-e67a-4a06-bb96-64b781e167c7") : configmap references non-existent config key: service-ca.crt
Jul 07 02:22:06.237126 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:06.237049    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 07 02:22:06.237126 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:06.237066    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs podName:e897bc4a-e67a-4a06-bb96-64b781e167c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:07.237055719 +0000 UTC m=+262.644645384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs") pod "router-default-996f5cb75-q2sjq" (UID: "e897bc4a-e67a-4a06-bb96-64b781e167c7") : secret "router-metrics-certs-default" not found
Jul 07 02:22:06.237126 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:06.237099    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls podName:40be13c4-b1af-4941-8d7b-3ad560fefb56 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:07.23708468 +0000 UTC m=+262.644674346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vjxfv" (UID: "40be13c4-b1af-4941-8d7b-3ad560fefb56") : secret "samples-operator-tls" not found
Jul 07 02:22:06.775773 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:06.775689    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-jlf4s" event={"ID":"a482dca7-e069-4636-aefe-bf75e2d4ed17","Type":"ContainerStarted","Data":"13e6d989e32b0d9f393c0547faf837eec7eb685072cdc3d6986051e5dab441cd"}
Jul 07 02:22:06.778223 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:06.778144    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf" event={"ID":"9f681b31-f05e-4f17-8b0d-7aaac08ea039","Type":"ContainerStarted","Data":"32d06d1539afe3afa87763065bab77460b4f4c96e389732be0a41e778a8310f5"}
Jul 07 02:22:06.784000 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:06.783922    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4" event={"ID":"f82c493c-8343-46b9-a50c-5dc1414d66f0","Type":"ContainerStarted","Data":"38a961de00f6e88c2a1091f5ba2b593a05085a21bfa9cbfe0a758b5bd9c07366"}
Jul 07 02:22:06.786744 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:06.786691    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256" event={"ID":"6b3022fc-1859-4338-bb06-cef488979501","Type":"ContainerStarted","Data":"23c1c93c9e430551166f008b549e5e1a544d39787a30922c13dd1d0d80f4f7b1"}
Jul 07 02:22:07.145681 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:07.145031    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:07.145681 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:07.145178    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:07.145681 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:07.145244    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls podName:a15d589e-a296-4ba9-83d9-2a8a74f8823e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:09.145223836 +0000 UTC m=+264.552813502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-t678z" (UID: "a15d589e-a296-4ba9-83d9-2a8a74f8823e") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:07.246674 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:07.245916    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:07.246674 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:07.246130    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 02:22:07.246674 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:07.246208    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs podName:e897bc4a-e67a-4a06-bb96-64b781e167c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:09.24618511 +0000 UTC m=+264.653774777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs") pod "router-default-996f5cb75-q2sjq" (UID: "e897bc4a-e67a-4a06-bb96-64b781e167c7") : secret "router-metrics-certs-default" not found
Jul 07 02:22:07.246674 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:07.246263    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:07.246674 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:07.246456    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle podName:e897bc4a-e67a-4a06-bb96-64b781e167c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:09.246437602 +0000 UTC m=+264.654027271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle") pod "router-default-996f5cb75-q2sjq" (UID: "e897bc4a-e67a-4a06-bb96-64b781e167c7") : configmap references non-existent config key: service-ca.crt
Jul 07 02:22:07.246674 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:07.246608    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 07 02:22:07.246674 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:07.246650    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls podName:40be13c4-b1af-4941-8d7b-3ad560fefb56 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:09.246636566 +0000 UTC m=+264.654226233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vjxfv" (UID: "40be13c4-b1af-4941-8d7b-3ad560fefb56") : secret "samples-operator-tls" not found
Jul 07 02:22:07.247166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:07.247008    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vjxfv\" (UID: \"40be13c4-b1af-4941-8d7b-3ad560fefb56\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:09.165381 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.165296    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:09.165700 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:09.165462    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:09.165700 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:09.165537    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls podName:a15d589e-a296-4ba9-83d9-2a8a74f8823e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:13.165517292 +0000 UTC m=+268.573106971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-t678z" (UID: "a15d589e-a296-4ba9-83d9-2a8a74f8823e") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:09.265706 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.265677    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vjxfv\" (UID: \"40be13c4-b1af-4941-8d7b-3ad560fefb56\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:09.265851 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.265721    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:09.265851 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.265802    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:09.265851 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:09.265809    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 07 02:22:09.265990 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:09.265856    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 02:22:09.265990 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:09.265861    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls podName:40be13c4-b1af-4941-8d7b-3ad560fefb56 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:13.265846562 +0000 UTC m=+268.673436229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vjxfv" (UID: "40be13c4-b1af-4941-8d7b-3ad560fefb56") : secret "samples-operator-tls" not found
Jul 07 02:22:09.265990 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:09.265916    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs podName:e897bc4a-e67a-4a06-bb96-64b781e167c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:13.265901682 +0000 UTC m=+268.673491349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs") pod "router-default-996f5cb75-q2sjq" (UID: "e897bc4a-e67a-4a06-bb96-64b781e167c7") : secret "router-metrics-certs-default" not found
Jul 07 02:22:09.265990 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:09.265931    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle podName:e897bc4a-e67a-4a06-bb96-64b781e167c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:13.265922152 +0000 UTC m=+268.673511823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle") pod "router-default-996f5cb75-q2sjq" (UID: "e897bc4a-e67a-4a06-bb96-64b781e167c7") : configmap references non-existent config key: service-ca.crt
Jul 07 02:22:09.797042 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.797004    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4" event={"ID":"f82c493c-8343-46b9-a50c-5dc1414d66f0","Type":"ContainerStarted","Data":"040a49416617f64cba7819da06afbba45a77cf2e9651c68ecdec2b33aea20f52"}
Jul 07 02:22:09.798581 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.798551    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256" event={"ID":"6b3022fc-1859-4338-bb06-cef488979501","Type":"ContainerStarted","Data":"802ac4a3713e13423eb3dd10157d72941c2da8f58b244660f5539fa0417e0ab2"}
Jul 07 02:22:09.800053 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.800030    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-jlf4s" event={"ID":"a482dca7-e069-4636-aefe-bf75e2d4ed17","Type":"ContainerStarted","Data":"125fff960680729f4bf37d8bc540cf4ab18eeb498fac67687e61eeac82a8929b"}
Jul 07 02:22:09.801469 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.801448    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf" event={"ID":"9f681b31-f05e-4f17-8b0d-7aaac08ea039","Type":"ContainerStarted","Data":"00e17b4cb39b3d817ed1db84be7506b9e467bf541f238560010f59ed31d965f8"}
Jul 07 02:22:09.814539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.814490    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4" podStartSLOduration=1.83640849 podStartE2EDuration="4.814474428s" totalImagesPullingTime="2.978065938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:09.813171709 +0000 UTC m=+265.220761397" watchObservedRunningTime="2026-07-07 02:22:09.814474428 +0000 UTC m=+265.222064116"
Jul 07 02:22:09.828742 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.828701    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256" podStartSLOduration=1.827715774 podStartE2EDuration="4.828685045s" totalImagesPullingTime="3.000969271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:09.828634112 +0000 UTC m=+265.236223800" watchObservedRunningTime="2026-07-07 02:22:09.828685045 +0000 UTC m=+265.236274733"
Jul 07 02:22:09.844989 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.844944    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf" podStartSLOduration=1.8323512499999999 podStartE2EDuration="4.844934889s" totalImagesPullingTime="3.012583639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:09.843945367 +0000 UTC m=+265.251535064" watchObservedRunningTime="2026-07-07 02:22:09.844934889 +0000 UTC m=+265.252524575"
Jul 07 02:22:09.858979 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:09.858943    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-jlf4s" podStartSLOduration=1.930423502 podStartE2EDuration="4.85893141s" totalImagesPullingTime="2.928507908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:09.858409773 +0000 UTC m=+265.265999462" watchObservedRunningTime="2026-07-07 02:22:09.85893141 +0000 UTC m=+265.266521096"
Jul 07 02:22:10.487022 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.486987    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vt8kn"]
Jul 07 02:22:10.490705 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.490686    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.495207 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.495187    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 02:22:10.495583 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.495319    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6hsdg\""
Jul 07 02:22:10.495583 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.495260    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 02:22:10.505167 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.505147    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vt8kn"]
Jul 07 02:22:10.578122 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.578096    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrn4q\" (UniqueName: \"kubernetes.io/projected/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-api-access-rrn4q\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.578210 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.578173    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-crio-socket\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.578210 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.578192    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-data-volume\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.578281 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.578236    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.578339 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.578322    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.678886 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.678861    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.678987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.678911    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.678987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.678968    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrn4q\" (UniqueName: \"kubernetes.io/projected/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-api-access-rrn4q\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.679095 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:10.678998    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:10.679095 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:10.679038    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:22:10.679095 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:10.679062    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls podName:edfa0e44-4c4e-4e5d-bc18-cfaf99348413 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:11.179042737 +0000 UTC m=+266.586632415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vt8kn" (UID: "edfa0e44-4c4e-4e5d-bc18-cfaf99348413") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:10.679243 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.679135    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-crio-socket\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.679243 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.679168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-data-volume\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.679243 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:10.679185    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm podName:edfa0e44-4c4e-4e5d-bc18-cfaf99348413 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:11.179163277 +0000 UTC m=+266.586752963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vt8kn" (UID: "edfa0e44-4c4e-4e5d-bc18-cfaf99348413") : configmap "kube-rbac-proxy" not found
Jul 07 02:22:10.679243 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.679223    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-crio-socket\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.679525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.679501    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-data-volume\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:10.687659 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:10.687639    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrn4q\" (UniqueName: \"kubernetes.io/projected/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-api-access-rrn4q\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:11.184173 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:11.184139    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:11.184375 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:11.184181    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:11.184375 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:11.184257    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:22:11.184375 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:11.184266    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:11.184375 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:11.184308    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm podName:edfa0e44-4c4e-4e5d-bc18-cfaf99348413 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:12.184291663 +0000 UTC m=+267.591881328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vt8kn" (UID: "edfa0e44-4c4e-4e5d-bc18-cfaf99348413") : configmap "kube-rbac-proxy" not found
Jul 07 02:22:11.184375 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:11.184321    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls podName:edfa0e44-4c4e-4e5d-bc18-cfaf99348413 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:12.184315299 +0000 UTC m=+267.591904965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vt8kn" (UID: "edfa0e44-4c4e-4e5d-bc18-cfaf99348413") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:11.725118 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:11.725086    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv"]
Jul 07 02:22:11.728410 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:11.728386    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv"
Jul 07 02:22:11.731212 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:11.731193    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 07 02:22:11.731321 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:11.731288    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 07 02:22:11.732273 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:11.732254    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vt6kl\""
Jul 07 02:22:11.736559 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:11.736535    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv"]
Jul 07 02:22:11.889737 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:11.889713    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6spl\" (UniqueName: \"kubernetes.io/projected/d29f7fe1-2c3b-432f-a5a1-868515bad25c-kube-api-access-g6spl\") pod \"migrator-d57d58cc9-bdrzv\" (UID: \"d29f7fe1-2c3b-432f-a5a1-868515bad25c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv"
Jul 07 02:22:11.990115 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:11.990052    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6spl\" (UniqueName: \"kubernetes.io/projected/d29f7fe1-2c3b-432f-a5a1-868515bad25c-kube-api-access-g6spl\") pod \"migrator-d57d58cc9-bdrzv\" (UID: \"d29f7fe1-2c3b-432f-a5a1-868515bad25c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv"
Jul 07 02:22:11.997877 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:11.997852    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6spl\" (UniqueName: \"kubernetes.io/projected/d29f7fe1-2c3b-432f-a5a1-868515bad25c-kube-api-access-g6spl\") pod \"migrator-d57d58cc9-bdrzv\" (UID: \"d29f7fe1-2c3b-432f-a5a1-868515bad25c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv"
Jul 07 02:22:12.038134 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:12.038103    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv"
Jul 07 02:22:12.160953 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:12.160925    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv"]
Jul 07 02:22:12.163699 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:22:12.163669    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29f7fe1_2c3b_432f_a5a1_868515bad25c.slice/crio-1504a4969b8c95214f6edfe151d999b169d3135bc1a14fe1752cb3310975a41c WatchSource:0}: Error finding container 1504a4969b8c95214f6edfe151d999b169d3135bc1a14fe1752cb3310975a41c: Status 404 returned error can't find the container with id 1504a4969b8c95214f6edfe151d999b169d3135bc1a14fe1752cb3310975a41c
Jul 07 02:22:12.192420 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:12.192399    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:12.192502 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:12.192446    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:12.192553 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:12.192537    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:12.192605 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:12.192591    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:22:12.192638 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:12.192598    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls podName:edfa0e44-4c4e-4e5d-bc18-cfaf99348413 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:14.192578623 +0000 UTC m=+269.600168295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vt8kn" (UID: "edfa0e44-4c4e-4e5d-bc18-cfaf99348413") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:12.192677 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:12.192650    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm podName:edfa0e44-4c4e-4e5d-bc18-cfaf99348413 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:14.1926335 +0000 UTC m=+269.600223167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vt8kn" (UID: "edfa0e44-4c4e-4e5d-bc18-cfaf99348413") : configmap "kube-rbac-proxy" not found
Jul 07 02:22:12.263835 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:12.263792    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h4v89_785f14dc-4626-4437-be21-2761e326fe7d/dns-node-resolver/0.log"
Jul 07 02:22:12.811879 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:12.811845    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv" event={"ID":"d29f7fe1-2c3b-432f-a5a1-868515bad25c","Type":"ContainerStarted","Data":"1504a4969b8c95214f6edfe151d999b169d3135bc1a14fe1752cb3310975a41c"}
Jul 07 02:22:13.203178 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:13.203085    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:13.203334 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:13.203262    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:13.203432 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:13.203337    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls podName:a15d589e-a296-4ba9-83d9-2a8a74f8823e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:21.203316964 +0000 UTC m=+276.610906628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-t678z" (UID: "a15d589e-a296-4ba9-83d9-2a8a74f8823e") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:13.304239 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:13.304203    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:13.304469 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:13.304312    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vjxfv\" (UID: \"40be13c4-b1af-4941-8d7b-3ad560fefb56\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:13.304469 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:13.304379    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:13.304469 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:13.304405    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle podName:e897bc4a-e67a-4a06-bb96-64b781e167c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:21.304383001 +0000 UTC m=+276.711972666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle") pod "router-default-996f5cb75-q2sjq" (UID: "e897bc4a-e67a-4a06-bb96-64b781e167c7") : configmap references non-existent config key: service-ca.crt
Jul 07 02:22:13.304469 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:13.304459    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 07 02:22:13.304469 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:13.304466    2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 07 02:22:13.304710 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:13.304509    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs podName:e897bc4a-e67a-4a06-bb96-64b781e167c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:21.304493453 +0000 UTC m=+276.712083117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs") pod "router-default-996f5cb75-q2sjq" (UID: "e897bc4a-e67a-4a06-bb96-64b781e167c7") : secret "router-metrics-certs-default" not found
Jul 07 02:22:13.304710 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:13.304528    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls podName:40be13c4-b1af-4941-8d7b-3ad560fefb56 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:21.30451929 +0000 UTC m=+276.712108961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-vjxfv" (UID: "40be13c4-b1af-4941-8d7b-3ad560fefb56") : secret "samples-operator-tls" not found
Jul 07 02:22:13.664445 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:13.664419    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x54gh_c7c7db16-1364-477c-8b49-489562b8683c/node-ca/0.log"
Jul 07 02:22:13.816801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:13.816772    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv" event={"ID":"d29f7fe1-2c3b-432f-a5a1-868515bad25c","Type":"ContainerStarted","Data":"3115ef8cbdb5044e2f29ba73083b6ff3301494cd392de399741888ac69516425"}
Jul 07 02:22:13.817193 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:13.816817    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv" event={"ID":"d29f7fe1-2c3b-432f-a5a1-868515bad25c","Type":"ContainerStarted","Data":"34ba48dc20d0abd460bdf548fec69c945b8410c4206872511395648b6001fd06"}
Jul 07 02:22:13.834877 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:13.834835    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bdrzv" podStartSLOduration=1.629881372 podStartE2EDuration="2.834822013s" totalImagesPullingTime="1.204940641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:13.834643273 +0000 UTC m=+269.242232961" watchObservedRunningTime="2026-07-07 02:22:13.834822013 +0000 UTC m=+269.242411700"
Jul 07 02:22:14.210463 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:14.210388    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:14.210594 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:14.210511    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:14.210594 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:14.210517    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:22:14.210594 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:14.210579    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm podName:edfa0e44-4c4e-4e5d-bc18-cfaf99348413 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:18.210559246 +0000 UTC m=+273.618148911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vt8kn" (UID: "edfa0e44-4c4e-4e5d-bc18-cfaf99348413") : configmap "kube-rbac-proxy" not found
Jul 07 02:22:14.210698 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:14.210595    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:14.210698 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:14.210637    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls podName:edfa0e44-4c4e-4e5d-bc18-cfaf99348413 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:18.210626434 +0000 UTC m=+273.618216098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vt8kn" (UID: "edfa0e44-4c4e-4e5d-bc18-cfaf99348413") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:14.464079 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:14.463993    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rdp5n_d9982460-1009-4f83-8cfd-165fe6308074/serve-healthcheck-canary/0.log"
Jul 07 02:22:18.243616 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:18.243575    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:18.243616 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:18.243626    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:18.244092 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:18.243692    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:22:18.244092 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:18.243738    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm podName:edfa0e44-4c4e-4e5d-bc18-cfaf99348413 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:26.243725362 +0000 UTC m=+281.651315040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vt8kn" (UID: "edfa0e44-4c4e-4e5d-bc18-cfaf99348413") : configmap "kube-rbac-proxy" not found
Jul 07 02:22:18.244092 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:18.243753    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:18.244092 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:18.243819    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls podName:edfa0e44-4c4e-4e5d-bc18-cfaf99348413 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:26.243801053 +0000 UTC m=+281.651390723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vt8kn" (UID: "edfa0e44-4c4e-4e5d-bc18-cfaf99348413") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:21.271097 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:21.271059    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:21.271590 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:21.271177    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:21.271590 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:21.271253    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls podName:a15d589e-a296-4ba9-83d9-2a8a74f8823e nodeName:}" failed. No retries permitted until 2026-07-07 02:22:37.271232238 +0000 UTC m=+292.678821904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-t678z" (UID: "a15d589e-a296-4ba9-83d9-2a8a74f8823e") : secret "cluster-monitoring-operator-tls" not found
Jul 07 02:22:21.372359 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:21.372318    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vjxfv\" (UID: \"40be13c4-b1af-4941-8d7b-3ad560fefb56\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:21.372499 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:21.372382    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:21.372499 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:21.372433    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:21.372612 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:21.372584    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle podName:e897bc4a-e67a-4a06-bb96-64b781e167c7 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:37.372569642 +0000 UTC m=+292.780159307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle") pod "router-default-996f5cb75-q2sjq" (UID: "e897bc4a-e67a-4a06-bb96-64b781e167c7") : configmap references non-existent config key: service-ca.crt
Jul 07 02:22:21.374909 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:21.374886    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e897bc4a-e67a-4a06-bb96-64b781e167c7-metrics-certs\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:21.375033 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:21.375014    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/40be13c4-b1af-4941-8d7b-3ad560fefb56-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-vjxfv\" (UID: \"40be13c4-b1af-4941-8d7b-3ad560fefb56\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:21.408036 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:21.408018    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"
Jul 07 02:22:21.532364 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:21.532316    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv"]
Jul 07 02:22:21.840847 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:21.840773    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv" event={"ID":"40be13c4-b1af-4941-8d7b-3ad560fefb56","Type":"ContainerStarted","Data":"c9cac4242839106dbbdf9bc3237b32cf6e69023419f3bacffed60876e2ee1e22"}
Jul 07 02:22:23.848519 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:23.848439    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv" event={"ID":"40be13c4-b1af-4941-8d7b-3ad560fefb56","Type":"ContainerStarted","Data":"893df5c69e5573b2a57bea6e14562e6ad69e800847d26662f6a2445d84719fd0"}
Jul 07 02:22:23.848519 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:23.848505    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv" event={"ID":"40be13c4-b1af-4941-8d7b-3ad560fefb56","Type":"ContainerStarted","Data":"4c79abd75acb6896e6530f431289a078092f48c39acd8fba0a506c5925920350"}
Jul 07 02:22:23.866194 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:23.866154    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-vjxfv" podStartSLOduration=16.878420979 podStartE2EDuration="18.866140823s" totalImagesPullingTime="1.987719844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:23.865289376 +0000 UTC m=+279.272879063" watchObservedRunningTime="2026-07-07 02:22:23.866140823 +0000 UTC m=+279.273730522"
Jul 07 02:22:26.314187 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:26.314152    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:26.314187 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:26.314191    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:26.314738 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:26.314708    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:26.316564 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:26.316542    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edfa0e44-4c4e-4e5d-bc18-cfaf99348413-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vt8kn\" (UID: \"edfa0e44-4c4e-4e5d-bc18-cfaf99348413\") " pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:26.399533 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:26.399506    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vt8kn"
Jul 07 02:22:26.527177 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:26.527136    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp" podUID="34317344-6095-46da-b8ee-00d49db5a486"
Jul 07 02:22:26.527177 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:26.527145    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hfblg" podUID="17e64b2d-eebf-40dc-a342-909eb26792ba"
Jul 07 02:22:26.527177 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:26.527145    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-66bff4b87-w97mk" podUID="0dd8bcac-f023-4c21-ad00-520136c355ac"
Jul 07 02:22:26.530211 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:26.530185    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vt8kn"]
Jul 07 02:22:26.533120 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:22:26.533093    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfa0e44_4c4e_4e5d_bc18_cfaf99348413.slice/crio-f80f6171f28d217ff29024672e227ebc4b7e7d5f740c700052f9119bb56eb452 WatchSource:0}: Error finding container f80f6171f28d217ff29024672e227ebc4b7e7d5f740c700052f9119bb56eb452: Status 404 returned error can't find the container with id f80f6171f28d217ff29024672e227ebc4b7e7d5f740c700052f9119bb56eb452
Jul 07 02:22:26.858555 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:26.858527    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:22:26.858555 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:26.858551    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:22:26.858763 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:26.858563    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hfblg"
Jul 07 02:22:26.858763 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:26.858564    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt8kn" event={"ID":"edfa0e44-4c4e-4e5d-bc18-cfaf99348413","Type":"ContainerStarted","Data":"f80f6171f28d217ff29024672e227ebc4b7e7d5f740c700052f9119bb56eb452"}
Jul 07 02:22:29.945418 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:29.945376    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:22:29.945880 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:29.945434    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:22:29.945880 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:29.945593    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:22:29.948218 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:29.948188    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"image-registry-66bff4b87-w97mk\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") " pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:22:29.948558 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:29.948537    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17e64b2d-eebf-40dc-a342-909eb26792ba-metrics-tls\") pod \"dns-default-hfblg\" (UID: \"17e64b2d-eebf-40dc-a342-909eb26792ba\") " pod="openshift-dns/dns-default-hfblg"
Jul 07 02:22:29.948690 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:29.948671    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/34317344-6095-46da-b8ee-00d49db5a486-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-ljlsp\" (UID: \"34317344-6095-46da-b8ee-00d49db5a486\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:22:30.163089 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.163071    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ll4bl\""
Jul 07 02:22:30.163172 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.163070    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l8w8v\""
Jul 07 02:22:30.163172 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.163071    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kkr4v\""
Jul 07 02:22:30.169760 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.169704    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hfblg"
Jul 07 02:22:30.169760 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.169728    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:22:30.169918 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.169805    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"
Jul 07 02:22:30.344768 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.344698    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66bff4b87-w97mk"]
Jul 07 02:22:30.349033 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:22:30.349010    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd8bcac_f023_4c21_ad00_520136c355ac.slice/crio-5fadac92f144e4c2dd298a2b88be708d0ff2b941d688c2ccbb80224b75510cf8 WatchSource:0}: Error finding container 5fadac92f144e4c2dd298a2b88be708d0ff2b941d688c2ccbb80224b75510cf8: Status 404 returned error can't find the container with id 5fadac92f144e4c2dd298a2b88be708d0ff2b941d688c2ccbb80224b75510cf8
Jul 07 02:22:30.357785 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.357761    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hfblg"]
Jul 07 02:22:30.361188 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:22:30.361162    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e64b2d_eebf_40dc_a342_909eb26792ba.slice/crio-c438c3e9ce8c75a699e721c3e42bea6d6a8ec90fce46086cd609fd62dc002c97 WatchSource:0}: Error finding container c438c3e9ce8c75a699e721c3e42bea6d6a8ec90fce46086cd609fd62dc002c97: Status 404 returned error can't find the container with id c438c3e9ce8c75a699e721c3e42bea6d6a8ec90fce46086cd609fd62dc002c97
Jul 07 02:22:30.373807 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.373788    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp"]
Jul 07 02:22:30.376406 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:22:30.376385    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34317344_6095_46da_b8ee_00d49db5a486.slice/crio-d140eced4f646c20d402f134dc8d5fb40e9b69d5c2e28984d24ada7e7aec441c WatchSource:0}: Error finding container d140eced4f646c20d402f134dc8d5fb40e9b69d5c2e28984d24ada7e7aec441c: Status 404 returned error can't find the container with id d140eced4f646c20d402f134dc8d5fb40e9b69d5c2e28984d24ada7e7aec441c
Jul 07 02:22:30.880368 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.880316    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt8kn" event={"ID":"edfa0e44-4c4e-4e5d-bc18-cfaf99348413","Type":"ContainerStarted","Data":"cb1450aae2b95963ac8d1235504d8c152bbc179bbfdf92b16096830f0027f42c"}
Jul 07 02:22:30.881557 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.881532    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp" event={"ID":"34317344-6095-46da-b8ee-00d49db5a486","Type":"ContainerStarted","Data":"d140eced4f646c20d402f134dc8d5fb40e9b69d5c2e28984d24ada7e7aec441c"}
Jul 07 02:22:30.883033 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.883006    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66bff4b87-w97mk" event={"ID":"0dd8bcac-f023-4c21-ad00-520136c355ac","Type":"ContainerStarted","Data":"1cceccc8884b5358046bdeeebe46d0db7e3c4565bd97948393b5eebc778463e4"}
Jul 07 02:22:30.883128 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.883043    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66bff4b87-w97mk" event={"ID":"0dd8bcac-f023-4c21-ad00-520136c355ac","Type":"ContainerStarted","Data":"5fadac92f144e4c2dd298a2b88be708d0ff2b941d688c2ccbb80224b75510cf8"}
Jul 07 02:22:30.883128 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.883087    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:22:30.884226 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.884201    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hfblg" event={"ID":"17e64b2d-eebf-40dc-a342-909eb26792ba","Type":"ContainerStarted","Data":"c438c3e9ce8c75a699e721c3e42bea6d6a8ec90fce46086cd609fd62dc002c97"}
Jul 07 02:22:30.927992 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:30.927952    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66bff4b87-w97mk" podStartSLOduration=285.927937014 podStartE2EDuration="4m45.927937014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:30.926912794 +0000 UTC m=+286.334502496" watchObservedRunningTime="2026-07-07 02:22:30.927937014 +0000 UTC m=+286.335526681"
Jul 07 02:22:31.891322 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:31.891291    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt8kn" event={"ID":"edfa0e44-4c4e-4e5d-bc18-cfaf99348413","Type":"ContainerStarted","Data":"fa8b1994c0f345c3ba14f627a903d1edb2ddbc3fccd831e0996ccdbead4e6e4c"}
Jul 07 02:22:32.896459 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:32.896420    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp" event={"ID":"34317344-6095-46da-b8ee-00d49db5a486","Type":"ContainerStarted","Data":"1525ea708013545a966f37190c835f80ad6c15f86cb5573ecfc4cbb560713a00"}
Jul 07 02:22:32.912527 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:32.912481    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-ljlsp" podStartSLOduration=264.420658996 podStartE2EDuration="4m25.9124646s" totalImagesPullingTime="1.491805604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:32.911263544 +0000 UTC m=+288.318853232" watchObservedRunningTime="2026-07-07 02:22:32.9124646 +0000 UTC m=+288.320054288"
Jul 07 02:22:34.904542 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:34.904507    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hfblg" event={"ID":"17e64b2d-eebf-40dc-a342-909eb26792ba","Type":"ContainerStarted","Data":"1bff5963a515f3a9730d850da9c15089ad0bb28603b444aaca93e93651380498"}
Jul 07 02:22:34.904926 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:34.904582    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hfblg" event={"ID":"17e64b2d-eebf-40dc-a342-909eb26792ba","Type":"ContainerStarted","Data":"59cb3dfc39072f2b6ab71b6d9b36dc36f47fb5ada52abde53891c139506dfd5f"}
Jul 07 02:22:34.904926 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:34.904620    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hfblg"
Jul 07 02:22:34.906520 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:34.906498    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vt8kn" event={"ID":"edfa0e44-4c4e-4e5d-bc18-cfaf99348413","Type":"ContainerStarted","Data":"57f833633283b43d346ca17b2af2c96cd7ca57d235ba6f5e3a63732f3bdf34de"}
Jul 07 02:22:34.921196 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:34.921138    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hfblg" podStartSLOduration=252.038841971 podStartE2EDuration="4m15.921126082s" totalImagesPullingTime="3.882284111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:34.919885729 +0000 UTC m=+290.327475418" watchObservedRunningTime="2026-07-07 02:22:34.921126082 +0000 UTC m=+290.328715770"
Jul 07 02:22:34.937510 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:34.937469    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vt8kn" podStartSLOduration=17.388345294 podStartE2EDuration="24.937452552s" totalImagesPullingTime="7.549107258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:22:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:34.937409383 +0000 UTC m=+290.344999072" watchObservedRunningTime="2026-07-07 02:22:34.937452552 +0000 UTC m=+290.345042241"
Jul 07 02:22:37.309325 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.309287    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:37.311944 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.311917    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a15d589e-a296-4ba9-83d9-2a8a74f8823e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-t678z\" (UID: \"a15d589e-a296-4ba9-83d9-2a8a74f8823e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:37.410548 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.410520    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:37.411049 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.411032    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e897bc4a-e67a-4a06-bb96-64b781e167c7-service-ca-bundle\") pod \"router-default-996f5cb75-q2sjq\" (UID: \"e897bc4a-e67a-4a06-bb96-64b781e167c7\") " pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:37.516007 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.515939    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"
Jul 07 02:22:37.613016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.612855    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:37.642742 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.642719    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z"]
Jul 07 02:22:37.645024 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:22:37.644983    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15d589e_a296_4ba9_83d9_2a8a74f8823e.slice/crio-8a067367b52c67e582ad9bff3b838d443a5a515a2c378c3a91e49e8f57f4a09f WatchSource:0}: Error finding container 8a067367b52c67e582ad9bff3b838d443a5a515a2c378c3a91e49e8f57f4a09f: Status 404 returned error can't find the container with id 8a067367b52c67e582ad9bff3b838d443a5a515a2c378c3a91e49e8f57f4a09f
Jul 07 02:22:37.742154 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.742125    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-996f5cb75-q2sjq"]
Jul 07 02:22:37.745199 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:22:37.745166    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode897bc4a_e67a_4a06_bb96_64b781e167c7.slice/crio-123ff723c2fe625bf4ce0b4853cc6c37545c1c3c3b40e8d11bf35db87dc07b34 WatchSource:0}: Error finding container 123ff723c2fe625bf4ce0b4853cc6c37545c1c3c3b40e8d11bf35db87dc07b34: Status 404 returned error can't find the container with id 123ff723c2fe625bf4ce0b4853cc6c37545c1c3c3b40e8d11bf35db87dc07b34
Jul 07 02:22:37.916984 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.916906    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-996f5cb75-q2sjq" event={"ID":"e897bc4a-e67a-4a06-bb96-64b781e167c7","Type":"ContainerStarted","Data":"419615d407c86be51c920d3bedcce1807e0673403781302fe0de0c08e32d2ad6"}
Jul 07 02:22:37.916984 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.916957    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-996f5cb75-q2sjq" event={"ID":"e897bc4a-e67a-4a06-bb96-64b781e167c7","Type":"ContainerStarted","Data":"123ff723c2fe625bf4ce0b4853cc6c37545c1c3c3b40e8d11bf35db87dc07b34"}
Jul 07 02:22:37.917929 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.917908    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z" event={"ID":"a15d589e-a296-4ba9-83d9-2a8a74f8823e","Type":"ContainerStarted","Data":"8a067367b52c67e582ad9bff3b838d443a5a515a2c378c3a91e49e8f57f4a09f"}
Jul 07 02:22:37.936243 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:37.936191    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-996f5cb75-q2sjq" podStartSLOduration=32.936178104 podStartE2EDuration="32.936178104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:37.935038276 +0000 UTC m=+293.342627968" watchObservedRunningTime="2026-07-07 02:22:37.936178104 +0000 UTC m=+293.343767791"
Jul 07 02:22:38.613229 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:38.613196    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:38.615572 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:38.615550    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:38.921257 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:38.921182    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:38.922464 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:38.922443    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-996f5cb75-q2sjq"
Jul 07 02:22:39.925203 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:39.925170    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z" event={"ID":"a15d589e-a296-4ba9-83d9-2a8a74f8823e","Type":"ContainerStarted","Data":"3c8ccfc207ffbca63c8e00d69a8d9c6bcdc3f6639411ce0405aa21b038049618"}
Jul 07 02:22:39.942258 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:39.942216    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-t678z" podStartSLOduration=32.727356317 podStartE2EDuration="34.942200577s" totalImagesPullingTime="2.21484426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:39.940278038 +0000 UTC m=+295.347867726" watchObservedRunningTime="2026-07-07 02:22:39.942200577 +0000 UTC m=+295.349790264"
Jul 07 02:22:40.082936 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:40.082898    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wcsrt" podUID="e49d21df-0b01-4845-ad69-b2ad83b93187"
Jul 07 02:22:44.911374 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:44.911328    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hfblg"
Jul 07 02:22:44.973262 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:44.973223    2570 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 02:22:50.174762 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:50.174720    2570 patch_prober.go:28] interesting pod/image-registry-66bff4b87-w97mk container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 07 02:22:50.177268 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:50.174786    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-66bff4b87-w97mk" podUID="0dd8bcac-f023-4c21-ad00-520136c355ac" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:22:51.895587 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:51.895561    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:22:53.081616 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:53.081583    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:22:53.873035 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:53.873011    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66bff4b87-w97mk"]
Jul 07 02:22:56.811462 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.811429    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6j2dr"]
Jul 07 02:22:56.814706 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.814689    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:56.819318 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.819297    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 02:22:56.819456 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.819364    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 02:22:56.819456 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.819398    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 02:22:56.819456 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.819429    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fpv7g\""
Jul 07 02:22:56.819626 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.819372    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 02:22:56.947856 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.947827    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/472fd877-0e2d-4edb-a37e-5e61acfff687-root\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:56.948005 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.947863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/472fd877-0e2d-4edb-a37e-5e61acfff687-metrics-client-ca\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:56.948005 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.947882    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srw95\" (UniqueName: \"kubernetes.io/projected/472fd877-0e2d-4edb-a37e-5e61acfff687-kube-api-access-srw95\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:56.948005 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.947970    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-tls\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:56.948005 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.948003    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:56.948132 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.948023    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/472fd877-0e2d-4edb-a37e-5e61acfff687-sys\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:56.948132 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.948038    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-textfile\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:56.948132 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.948084    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-wtmp\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:56.948219 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:56.948137    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-accelerators-collector-config\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049022 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.048995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-accelerators-collector-config\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049144 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049025    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/472fd877-0e2d-4edb-a37e-5e61acfff687-root\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049144 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049049    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/472fd877-0e2d-4edb-a37e-5e61acfff687-metrics-client-ca\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049144 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049066    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srw95\" (UniqueName: \"kubernetes.io/projected/472fd877-0e2d-4edb-a37e-5e61acfff687-kube-api-access-srw95\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049144 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049119    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-tls\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049395 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049152    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049395 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049117    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/472fd877-0e2d-4edb-a37e-5e61acfff687-root\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049395 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049180    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/472fd877-0e2d-4edb-a37e-5e61acfff687-sys\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049395 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049217    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/472fd877-0e2d-4edb-a37e-5e61acfff687-sys\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049395 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049224    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-textfile\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049395 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:57.049251    2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 02:22:57.049395 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:22:57.049325    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-tls podName:472fd877-0e2d-4edb-a37e-5e61acfff687 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:57.549306582 +0000 UTC m=+312.956896265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-tls") pod "node-exporter-6j2dr" (UID: "472fd877-0e2d-4edb-a37e-5e61acfff687") : secret "node-exporter-tls" not found
Jul 07 02:22:57.049395 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049257    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-wtmp\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049395 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049388    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-wtmp\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049876 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049530    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-textfile\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049876 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049799    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/472fd877-0e2d-4edb-a37e-5e61acfff687-metrics-client-ca\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.049876 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.049797    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-accelerators-collector-config\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.051682 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.051658    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.059866 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.059837    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srw95\" (UniqueName: \"kubernetes.io/projected/472fd877-0e2d-4edb-a37e-5e61acfff687-kube-api-access-srw95\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.557590 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.553896    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-tls\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.559881 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.559853    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/472fd877-0e2d-4edb-a37e-5e61acfff687-node-exporter-tls\") pod \"node-exporter-6j2dr\" (UID: \"472fd877-0e2d-4edb-a37e-5e61acfff687\") " pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.723580 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.723557    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6j2dr"
Jul 07 02:22:57.735650 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:22:57.735628    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472fd877_0e2d_4edb_a37e_5e61acfff687.slice/crio-60d63167cec99e4ed35c33a432bd3437ae9c35aaaa0fcfe238e635cc8924dd0a WatchSource:0}: Error finding container 60d63167cec99e4ed35c33a432bd3437ae9c35aaaa0fcfe238e635cc8924dd0a: Status 404 returned error can't find the container with id 60d63167cec99e4ed35c33a432bd3437ae9c35aaaa0fcfe238e635cc8924dd0a
Jul 07 02:22:57.738212 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.738190    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:22:57.896216 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.896127    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:22:57.900886 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.900864    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:57.903707 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.903450    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 02:22:57.903707 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.903485    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5cp5g\""
Jul 07 02:22:57.903707 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.903502    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 02:22:57.903707 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.903454    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 02:22:57.903707 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.903694    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 02:22:57.904016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.903723    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 02:22:57.904016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.903763    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 02:22:57.904016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.903910    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 02:22:57.904276 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.904210    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 02:22:57.904602 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.904583    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 02:22:57.917097 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.917077    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:22:57.991303 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:57.991275    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6j2dr" event={"ID":"472fd877-0e2d-4edb-a37e-5e61acfff687","Type":"ContainerStarted","Data":"60d63167cec99e4ed35c33a432bd3437ae9c35aaaa0fcfe238e635cc8924dd0a"}
Jul 07 02:22:58.057683 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.057650    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-web-config\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.057784 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.057693    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.057784 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.057722    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-volume\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.057870 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.057788    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.057870 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.057836    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.057940 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.057870    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.057974 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.057952    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.058007 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.057979    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.058049 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.058015    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.058086 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.058067    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.058133 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.058085    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.058133 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.058107    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr6gl\" (UniqueName: \"kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-kube-api-access-xr6gl\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.058230 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.058174    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-out\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.158865 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.158811    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.158865 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.158837    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.158865 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.158853    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.158865 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.158867    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr6gl\" (UniqueName: \"kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-kube-api-access-xr6gl\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.159121 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.158890    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-out\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.159121 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.158916    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-web-config\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.159227 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.159121    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.159282 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.159251    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-volume\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.159334 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.159298    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.159334 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.159327    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.159493 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.159381    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.159493 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.159477    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.159597 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.159506    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.159755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.159732    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.160368 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.160329    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.160918 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.160898    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.161953 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.161923    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-out\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.162611 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.162283    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-web-config\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.162611 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.162581    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.162897 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.162875    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.163010 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.162990    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.163429 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.163407    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.164009 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.163991    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.164564 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.164546    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-volume\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.164620 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.164542    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.167426 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.167407    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr6gl\" (UniqueName: \"kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-kube-api-access-xr6gl\") pod \"alertmanager-main-0\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.211210 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.211189    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:22:58.365961 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.365929    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:22:58.369639 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:22:58.369608    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ff97e9_80a6_4cb8_97f3_42b343309ecd.slice/crio-512cbb01f030671fdc2d9b106575b0811ac963bdb90b49de63ac3b1508e9d4bf WatchSource:0}: Error finding container 512cbb01f030671fdc2d9b106575b0811ac963bdb90b49de63ac3b1508e9d4bf: Status 404 returned error can't find the container with id 512cbb01f030671fdc2d9b106575b0811ac963bdb90b49de63ac3b1508e9d4bf
Jul 07 02:22:58.996469 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.996440    2570 generic.go:356] "Generic (PLEG): container finished" podID="472fd877-0e2d-4edb-a37e-5e61acfff687" containerID="34d896235468342f820a84745e8759abbb48290d89555eff652ee7d49bde552b" exitCode=0
Jul 07 02:22:58.996860 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.996514    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6j2dr" event={"ID":"472fd877-0e2d-4edb-a37e-5e61acfff687","Type":"ContainerDied","Data":"34d896235468342f820a84745e8759abbb48290d89555eff652ee7d49bde552b"}
Jul 07 02:22:58.997743 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:58.997722    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerStarted","Data":"512cbb01f030671fdc2d9b106575b0811ac963bdb90b49de63ac3b1508e9d4bf"}
Jul 07 02:22:59.777820 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.777798    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-648d4ff86-ln78g"]
Jul 07 02:22:59.781311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.781294    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.784831 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.784728    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-chh34uk1rouh6\""
Jul 07 02:22:59.784831 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.784741    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 07 02:22:59.784831 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.784788    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 07 02:22:59.784831 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.784800    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xtzlq\""
Jul 07 02:22:59.784831 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.784788    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 07 02:22:59.784831 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.784791    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 07 02:22:59.785377 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.785337    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 07 02:22:59.801311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.798909    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-648d4ff86-ln78g"]
Jul 07 02:22:59.874523 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.874500    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-tls\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.874641 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.874543    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.874641 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.874562    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.874726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.874663    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.874726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.874697    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fmpq\" (UniqueName: \"kubernetes.io/projected/564204fd-ca9d-43c4-b1d9-0c546f8202e8-kube-api-access-9fmpq\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.874726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.874717    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-grpc-tls\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.874846 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.874738    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/564204fd-ca9d-43c4-b1d9-0c546f8202e8-metrics-client-ca\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.874846 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.874764    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.975866 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.975798    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.975866 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.975834    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.976042 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.975965    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.976042 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.976018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fmpq\" (UniqueName: \"kubernetes.io/projected/564204fd-ca9d-43c4-b1d9-0c546f8202e8-kube-api-access-9fmpq\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.976103 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.976042    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-grpc-tls\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.976103 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.976068    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/564204fd-ca9d-43c4-b1d9-0c546f8202e8-metrics-client-ca\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.976103 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.976088    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.976235 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.976133    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-tls\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.976862 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.976836    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/564204fd-ca9d-43c4-b1d9-0c546f8202e8-metrics-client-ca\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.979030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.978988    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.979141 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.979096    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.979217 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.979205    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.979274 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.979256    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-grpc-tls\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.979326 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.979256    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-tls\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.979326 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.979282    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/564204fd-ca9d-43c4-b1d9-0c546f8202e8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:22:59.990900 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:22:59.990876    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fmpq\" (UniqueName: \"kubernetes.io/projected/564204fd-ca9d-43c4-b1d9-0c546f8202e8-kube-api-access-9fmpq\") pod \"thanos-querier-648d4ff86-ln78g\" (UID: \"564204fd-ca9d-43c4-b1d9-0c546f8202e8\") " pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:23:00.002683 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:00.002651    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6j2dr" event={"ID":"472fd877-0e2d-4edb-a37e-5e61acfff687","Type":"ContainerStarted","Data":"71376f63f569dee8eda3174701165020b926bcc11c39bb4ce7b7852b489d93c1"}
Jul 07 02:23:00.003064 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:00.002705    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6j2dr" event={"ID":"472fd877-0e2d-4edb-a37e-5e61acfff687","Type":"ContainerStarted","Data":"4ec316764481b2f9578ffbd3c3ae10e9b6a5378163a8b946eced2591182b8184"}
Jul 07 02:23:00.004184 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:00.004162    2570 generic.go:356] "Generic (PLEG): container finished" podID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerID="5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e" exitCode=0
Jul 07 02:23:00.004289 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:00.004235    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerDied","Data":"5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e"}
Jul 07 02:23:00.025207 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:00.025169    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6j2dr" podStartSLOduration=2.94276725 podStartE2EDuration="4.025154802s" totalImagesPullingTime="1.012501891s" totalInitContainerRuntime="69.885661ms" isStatefulPod=true podCreationTimestamp="2026-07-07 02:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:23:00.023945812 +0000 UTC m=+315.431535503" watchObservedRunningTime="2026-07-07 02:23:00.025154802 +0000 UTC m=+315.432744489"
Jul 07 02:23:00.089977 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:00.089953    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:23:00.222898 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:00.222875    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-648d4ff86-ln78g"]
Jul 07 02:23:00.225076 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:23:00.225052    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564204fd_ca9d_43c4_b1d9_0c546f8202e8.slice/crio-2a1de189cd6bc27f5c20e7fd1b7de5ce9a518b2fb466d4ca8542be5b25dc5c9f WatchSource:0}: Error finding container 2a1de189cd6bc27f5c20e7fd1b7de5ce9a518b2fb466d4ca8542be5b25dc5c9f: Status 404 returned error can't find the container with id 2a1de189cd6bc27f5c20e7fd1b7de5ce9a518b2fb466d4ca8542be5b25dc5c9f
Jul 07 02:23:01.011031 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.010989    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g" event={"ID":"564204fd-ca9d-43c4-b1d9-0c546f8202e8","Type":"ContainerStarted","Data":"2a1de189cd6bc27f5c20e7fd1b7de5ce9a518b2fb466d4ca8542be5b25dc5c9f"}
Jul 07 02:23:01.146368 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.146323    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d85c655c5-kgzgq"]
Jul 07 02:23:01.154685 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.154660    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.157552 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.157528    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 07 02:23:01.157696 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.157631    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 07 02:23:01.158288 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.158158    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 07 02:23:01.158413 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.158364    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ttsvp\""
Jul 07 02:23:01.158413 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.158382    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c4erc1k3mt8v7\""
Jul 07 02:23:01.158533 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.158416    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 07 02:23:01.161860 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.161833    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d85c655c5-kgzgq"]
Jul 07 02:23:01.287411 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.287380    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22b133c-7d55-4adc-9344-49eade940c2e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.287588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.287426    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcvp\" (UniqueName: \"kubernetes.io/projected/a22b133c-7d55-4adc-9344-49eade940c2e-kube-api-access-kfcvp\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.287588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.287490    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a22b133c-7d55-4adc-9344-49eade940c2e-secret-metrics-server-client-certs\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.287588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.287521    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a22b133c-7d55-4adc-9344-49eade940c2e-secret-metrics-server-tls\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.287588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.287543    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a22b133c-7d55-4adc-9344-49eade940c2e-audit-log\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.287588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.287573    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a22b133c-7d55-4adc-9344-49eade940c2e-metrics-server-audit-profiles\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.287824 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.287601    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a22b133c-7d55-4adc-9344-49eade940c2e-client-ca-bundle\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.388650 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.388623    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22b133c-7d55-4adc-9344-49eade940c2e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.388650 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.388654    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcvp\" (UniqueName: \"kubernetes.io/projected/a22b133c-7d55-4adc-9344-49eade940c2e-kube-api-access-kfcvp\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.388846 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.388690    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a22b133c-7d55-4adc-9344-49eade940c2e-secret-metrics-server-client-certs\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.388846 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.388714    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a22b133c-7d55-4adc-9344-49eade940c2e-secret-metrics-server-tls\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.388846 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.388738    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a22b133c-7d55-4adc-9344-49eade940c2e-audit-log\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.388846 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.388773    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a22b133c-7d55-4adc-9344-49eade940c2e-metrics-server-audit-profiles\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.388846 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.388800    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a22b133c-7d55-4adc-9344-49eade940c2e-client-ca-bundle\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.389317 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.389275    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a22b133c-7d55-4adc-9344-49eade940c2e-audit-log\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.389496 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.389476    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22b133c-7d55-4adc-9344-49eade940c2e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.390307 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.390283    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a22b133c-7d55-4adc-9344-49eade940c2e-metrics-server-audit-profiles\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.392021 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.391959    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a22b133c-7d55-4adc-9344-49eade940c2e-secret-metrics-server-client-certs\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.392021 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.391983    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a22b133c-7d55-4adc-9344-49eade940c2e-secret-metrics-server-tls\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.392419 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.392395    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a22b133c-7d55-4adc-9344-49eade940c2e-client-ca-bundle\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.397557 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.397535    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcvp\" (UniqueName: \"kubernetes.io/projected/a22b133c-7d55-4adc-9344-49eade940c2e-kube-api-access-kfcvp\") pod \"metrics-server-d85c655c5-kgzgq\" (UID: \"a22b133c-7d55-4adc-9344-49eade940c2e\") " pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.468546 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.468519    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:01.634220 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.634193    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d85c655c5-kgzgq"]
Jul 07 02:23:01.779046 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:23:01.779012    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22b133c_7d55_4adc_9344_49eade940c2e.slice/crio-d37b4b98711cbf9098927308e5b7d0c8a54f6b4c5a7c2cc8e8b252f0316813e2 WatchSource:0}: Error finding container d37b4b98711cbf9098927308e5b7d0c8a54f6b4c5a7c2cc8e8b252f0316813e2: Status 404 returned error can't find the container with id d37b4b98711cbf9098927308e5b7d0c8a54f6b4c5a7c2cc8e8b252f0316813e2
Jul 07 02:23:01.784695 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.784667    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64d5898fc6-9z8vr"]
Jul 07 02:23:01.815291 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.815269    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64d5898fc6-9z8vr"]
Jul 07 02:23:01.815431 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.815384    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:01.818210 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.818188    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bvcr9\""
Jul 07 02:23:01.818753 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.818508    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 07 02:23:01.818753 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.818543    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 02:23:01.818753 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.818590    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 07 02:23:01.819085 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.819065    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 07 02:23:01.819500 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.819449    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 07 02:23:01.819500 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.819450    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 07 02:23:01.819633 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.819553    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 02:23:01.836378 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.833007    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 07 02:23:01.995098 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.995067    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-oauth-serving-cert\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:01.995230 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.995135    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4l8\" (UniqueName: \"kubernetes.io/projected/b634e258-4ecc-41f0-a799-47781f098c8d-kube-api-access-nd4l8\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:01.995230 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.995176    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-console-config\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:01.995230 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.995206    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-serving-cert\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:01.995422 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.995229    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-trusted-ca-bundle\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:01.995422 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.995261    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-service-ca\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:01.995422 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:01.995305    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-oauth-config\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.015899 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.015855    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq" event={"ID":"a22b133c-7d55-4adc-9344-49eade940c2e","Type":"ContainerStarted","Data":"d37b4b98711cbf9098927308e5b7d0c8a54f6b4c5a7c2cc8e8b252f0316813e2"}
Jul 07 02:23:02.018140 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.018117    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerStarted","Data":"981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e"}
Jul 07 02:23:02.096367 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.096282    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4l8\" (UniqueName: \"kubernetes.io/projected/b634e258-4ecc-41f0-a799-47781f098c8d-kube-api-access-nd4l8\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.096367 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.096332    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-console-config\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.096545 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.096381    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-serving-cert\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.096545 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.096404    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-trusted-ca-bundle\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.096545 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.096433    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-service-ca\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.096545 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.096487    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-oauth-config\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.096545 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.096524    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-oauth-serving-cert\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.097324 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.097251    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-console-config\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.097324 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.097257    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-service-ca\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.097324 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.097373    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-oauth-serving-cert\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.097600 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.097523    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-trusted-ca-bundle\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.099112 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.099093    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-oauth-config\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.099264 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.099242    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-serving-cert\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.105717 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.105695    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4l8\" (UniqueName: \"kubernetes.io/projected/b634e258-4ecc-41f0-a799-47781f098c8d-kube-api-access-nd4l8\") pod \"console-64d5898fc6-9z8vr\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") " pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.126643 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.126615    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:02.582808 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:02.582782    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64d5898fc6-9z8vr"]
Jul 07 02:23:02.586149 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:23:02.586120    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb634e258_4ecc_41f0_a799_47781f098c8d.slice/crio-71397286f388404e5bbf2fa5963bce621b2d5636878a29f9b974af30d60d8dc9 WatchSource:0}: Error finding container 71397286f388404e5bbf2fa5963bce621b2d5636878a29f9b974af30d60d8dc9: Status 404 returned error can't find the container with id 71397286f388404e5bbf2fa5963bce621b2d5636878a29f9b974af30d60d8dc9
Jul 07 02:23:03.026506 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.026474    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerStarted","Data":"9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f"}
Jul 07 02:23:03.026871 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.026567    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerStarted","Data":"d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34"}
Jul 07 02:23:03.026871 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.026614    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerStarted","Data":"76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f"}
Jul 07 02:23:03.028949 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.028923    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g" event={"ID":"564204fd-ca9d-43c4-b1d9-0c546f8202e8","Type":"ContainerStarted","Data":"d345eb8fe885150da5522abf6fe2b5e16490cf3ae64b40fad7cb60c70f158b1f"}
Jul 07 02:23:03.031148 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.031088    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d5898fc6-9z8vr" event={"ID":"b634e258-4ecc-41f0-a799-47781f098c8d","Type":"ContainerStarted","Data":"71397286f388404e5bbf2fa5963bce621b2d5636878a29f9b974af30d60d8dc9"}
Jul 07 02:23:03.039217 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.039149    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:23:03.056904 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.056876    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.064902 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.063805    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:23:03.064902 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.064170    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 02:23:03.064902 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.064388    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 02:23:03.064902 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.064557    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 02:23:03.064902 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.064611    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kv6z7\""
Jul 07 02:23:03.064902 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.064724    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 02:23:03.064902 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.064796    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 02:23:03.064902 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.064877    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 02:23:03.066258 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.065424    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 02:23:03.066258 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.065645    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dkhanorfma7ch\""
Jul 07 02:23:03.067571 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.067138    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 02:23:03.067571 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.067182    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 02:23:03.067571 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.067390    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 02:23:03.067571 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.067450    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 02:23:03.071599 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.068872    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 02:23:03.071599 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.069781    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 02:23:03.072484 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.072135    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 02:23:03.206413 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206177    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206413 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206413 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206275    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206413 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206328    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206432    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206466    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206523    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206558    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206582    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206608    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206633    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-config\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206667    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-config-out\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206691    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.206726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206718    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.207146 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206740    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb9qf\" (UniqueName: \"kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-kube-api-access-pb9qf\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.207146 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206770    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-web-config\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.207146 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206794    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.207146 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206842    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.207146 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206871    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.207146 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.206895    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308275 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308240    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308463 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308279    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308463 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308306    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308463 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308381    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308463 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308412    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308463 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308448    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308733 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308473    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308733 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308499    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308733 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308526    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308733 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308563    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308733 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308591    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308733 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308611    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308733 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308637    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308733 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308665    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-config\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.308733 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308712    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-config-out\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.309156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308738    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.309156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308767    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.309156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308790    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb9qf\" (UniqueName: \"kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-kube-api-access-pb9qf\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.309156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308826    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-web-config\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.309156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308851    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.309156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.308968    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.313002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.310744    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.313002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.311530    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.313002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.311781    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.313002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.311824    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.313002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.311842    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.313002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.312305    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.313002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.312411    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.313002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.312797    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.313002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.312891    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.314133 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.313855    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.314235 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.314165    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.314235 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.314207    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.314553 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.314528    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.315386 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.315362    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.315505 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.315481    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-web-config\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.315621 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.315553    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.316318 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.316299    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-config\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.316635 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.316615    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-config-out\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.322065 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.322043    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb9qf\" (UniqueName: \"kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-kube-api-access-pb9qf\") pod \"prometheus-k8s-0\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.374016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.373962    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:03.568935 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:03.568843    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:23:03.983625 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:23:03.983590    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51384e2b_4f02_42f0_a4b0_b99a21730f94.slice/crio-d971b861cece61f110fd34eeea224f18d20ae3d3bf619a1af366d43a5b3dc4bd WatchSource:0}: Error finding container d971b861cece61f110fd34eeea224f18d20ae3d3bf619a1af366d43a5b3dc4bd: Status 404 returned error can't find the container with id d971b861cece61f110fd34eeea224f18d20ae3d3bf619a1af366d43a5b3dc4bd
Jul 07 02:23:04.036896 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:04.036871    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerStarted","Data":"d971b861cece61f110fd34eeea224f18d20ae3d3bf619a1af366d43a5b3dc4bd"}
Jul 07 02:23:04.041056 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:04.041010    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerStarted","Data":"5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e"}
Jul 07 02:23:04.044010 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:04.043982    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g" event={"ID":"564204fd-ca9d-43c4-b1d9-0c546f8202e8","Type":"ContainerStarted","Data":"749c63d41c157be79d56dfac3d533fae2a44dbffec89572463fd546296b6e026"}
Jul 07 02:23:04.044115 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:04.044055    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g" event={"ID":"564204fd-ca9d-43c4-b1d9-0c546f8202e8","Type":"ContainerStarted","Data":"4fb27309b755aa094ff4a6335485bbb4821e17a9b527327d498955f2c51348a4"}
Jul 07 02:23:05.050320 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:05.050273    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq" event={"ID":"a22b133c-7d55-4adc-9344-49eade940c2e","Type":"ContainerStarted","Data":"3cb30395150bc11f4afd00de76fd3f0e3492e8e9610b0601b46b471cc40f61d6"}
Jul 07 02:23:05.052034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:05.052000    2570 generic.go:356] "Generic (PLEG): container finished" podID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerID="6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773" exitCode=0
Jul 07 02:23:05.052187 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:05.052087    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerDied","Data":"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"}
Jul 07 02:23:05.069840 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:05.069791    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq" podStartSLOduration=1.6107298700000001 podStartE2EDuration="4.069774813s" totalImagesPullingTime="2.459044943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:23:05.067440917 +0000 UTC m=+320.475030604" watchObservedRunningTime="2026-07-07 02:23:05.069774813 +0000 UTC m=+320.477364502"
Jul 07 02:23:07.064787 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:07.064750    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerStarted","Data":"abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409"}
Jul 07 02:23:07.068364 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:07.068316    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g" event={"ID":"564204fd-ca9d-43c4-b1d9-0c546f8202e8","Type":"ContainerStarted","Data":"0b6664219b76c45e64abdda64305d5d876de95b317646e1c03deed11b01a0fee"}
Jul 07 02:23:07.068493 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:07.068410    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g" event={"ID":"564204fd-ca9d-43c4-b1d9-0c546f8202e8","Type":"ContainerStarted","Data":"1c45b0543799ac2fcfc76fb75ba7a469d46f46be0eacde9aef4716db97a542b0"}
Jul 07 02:23:07.068493 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:07.068455    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g" event={"ID":"564204fd-ca9d-43c4-b1d9-0c546f8202e8","Type":"ContainerStarted","Data":"abf67006e8c40ef09ebfc31d4e264e50644534d3b359a65e81e2105ab934877c"}
Jul 07 02:23:07.068599 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:07.068500    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:23:07.070093 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:07.070068    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d5898fc6-9z8vr" event={"ID":"b634e258-4ecc-41f0-a799-47781f098c8d","Type":"ContainerStarted","Data":"b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c"}
Jul 07 02:23:07.093960 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:07.093910    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.991249674 podStartE2EDuration="10.0938983s" totalImagesPullingTime="6.086862454s" totalInitContainerRuntime="15.786172ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:23:07.092233576 +0000 UTC m=+322.499823275" watchObservedRunningTime="2026-07-07 02:23:07.0938983 +0000 UTC m=+322.501487986"
Jul 07 02:23:07.132021 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:07.131964    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g" podStartSLOduration=2.683295459 podStartE2EDuration="8.131951298s" totalImagesPullingTime="5.448655839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:23:07.129643222 +0000 UTC m=+322.537232911" watchObservedRunningTime="2026-07-07 02:23:07.131951298 +0000 UTC m=+322.539541011"
Jul 07 02:23:07.174078 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:07.174031    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64d5898fc6-9z8vr" podStartSLOduration=2.7150192459999998 podStartE2EDuration="6.174016305s" totalImagesPullingTime="3.458997059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:23:07.173160952 +0000 UTC m=+322.580750639" watchObservedRunningTime="2026-07-07 02:23:07.174016305 +0000 UTC m=+322.581605991"
Jul 07 02:23:08.081319 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:08.081096    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-648d4ff86-ln78g"
Jul 07 02:23:10.089761 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:10.089729    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerStarted","Data":"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"}
Jul 07 02:23:10.090156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:10.089790    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerStarted","Data":"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"}
Jul 07 02:23:10.090156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:10.089816    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerStarted","Data":"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"}
Jul 07 02:23:10.090156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:10.089840    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerStarted","Data":"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"}
Jul 07 02:23:10.090156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:10.089864    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerStarted","Data":"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"}
Jul 07 02:23:10.090156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:10.089887    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerStarted","Data":"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"}
Jul 07 02:23:10.120215 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:10.120151    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.96974877 podStartE2EDuration="7.120135581s" totalImagesPullingTime="4.121501794s" totalInitContainerRuntime="28.885017ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:23:10.118580556 +0000 UTC m=+325.526170269" watchObservedRunningTime="2026-07-07 02:23:10.120135581 +0000 UTC m=+325.527725267"
Jul 07 02:23:12.127151 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:12.127118    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:12.127575 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:12.127176    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:12.133311 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:12.133288    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:13.105551 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:13.105523    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:23:13.374850 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:13.374775    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:23:18.898785 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:18.898744    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66bff4b87-w97mk" podUID="0dd8bcac-f023-4c21-ad00-520136c355ac" containerName="registry" containerID="cri-o://1cceccc8884b5358046bdeeebe46d0db7e3c4565bd97948393b5eebc778463e4" gracePeriod=30
Jul 07 02:23:19.128400 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.128371    2570 generic.go:356] "Generic (PLEG): container finished" podID="0dd8bcac-f023-4c21-ad00-520136c355ac" containerID="1cceccc8884b5358046bdeeebe46d0db7e3c4565bd97948393b5eebc778463e4" exitCode=0
Jul 07 02:23:19.128514 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.128447    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66bff4b87-w97mk" event={"ID":"0dd8bcac-f023-4c21-ad00-520136c355ac","Type":"ContainerDied","Data":"1cceccc8884b5358046bdeeebe46d0db7e3c4565bd97948393b5eebc778463e4"}
Jul 07 02:23:19.128514 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.128496    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66bff4b87-w97mk" event={"ID":"0dd8bcac-f023-4c21-ad00-520136c355ac","Type":"ContainerDied","Data":"5fadac92f144e4c2dd298a2b88be708d0ff2b941d688c2ccbb80224b75510cf8"}
Jul 07 02:23:19.128514 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.128512    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fadac92f144e4c2dd298a2b88be708d0ff2b941d688c2ccbb80224b75510cf8"
Jul 07 02:23:19.137573 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.137557    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:23:19.156466 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.156416    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-image-registry-private-configuration\") pod \"0dd8bcac-f023-4c21-ad00-520136c355ac\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") "
Jul 07 02:23:19.156466 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.156461    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0dd8bcac-f023-4c21-ad00-520136c355ac-ca-trust-extracted\") pod \"0dd8bcac-f023-4c21-ad00-520136c355ac\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") "
Jul 07 02:23:19.156634 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.156494    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") pod \"0dd8bcac-f023-4c21-ad00-520136c355ac\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") "
Jul 07 02:23:19.156634 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.156518    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-certificates\") pod \"0dd8bcac-f023-4c21-ad00-520136c355ac\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") "
Jul 07 02:23:19.156634 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.156595    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76gqn\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-kube-api-access-76gqn\") pod \"0dd8bcac-f023-4c21-ad00-520136c355ac\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") "
Jul 07 02:23:19.156784 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.156635    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-bound-sa-token\") pod \"0dd8bcac-f023-4c21-ad00-520136c355ac\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") "
Jul 07 02:23:19.156784 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.156667    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-trusted-ca\") pod \"0dd8bcac-f023-4c21-ad00-520136c355ac\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") "
Jul 07 02:23:19.156784 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.156694    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-installation-pull-secrets\") pod \"0dd8bcac-f023-4c21-ad00-520136c355ac\" (UID: \"0dd8bcac-f023-4c21-ad00-520136c355ac\") "
Jul 07 02:23:19.157070 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.157044    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0dd8bcac-f023-4c21-ad00-520136c355ac" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:23:19.157161 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.157137    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0dd8bcac-f023-4c21-ad00-520136c355ac" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:23:19.159088 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.159061    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0dd8bcac-f023-4c21-ad00-520136c355ac" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:23:19.159434 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.159386    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-kube-api-access-76gqn" (OuterVolumeSpecName: "kube-api-access-76gqn") pod "0dd8bcac-f023-4c21-ad00-520136c355ac" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac"). InnerVolumeSpecName "kube-api-access-76gqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:23:19.159525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.159506    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0dd8bcac-f023-4c21-ad00-520136c355ac" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:23:19.159582 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.159545    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0dd8bcac-f023-4c21-ad00-520136c355ac" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:23:19.159659 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.159635    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0dd8bcac-f023-4c21-ad00-520136c355ac" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:23:19.167111 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.167091    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd8bcac-f023-4c21-ad00-520136c355ac-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0dd8bcac-f023-4c21-ad00-520136c355ac" (UID: "0dd8bcac-f023-4c21-ad00-520136c355ac"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:23:19.257699 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.257664    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76gqn\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-kube-api-access-76gqn\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:23:19.257699 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.257700    2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-bound-sa-token\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:23:19.257874 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.257709    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-trusted-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:23:19.257874 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.257717    2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-installation-pull-secrets\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:23:19.257874 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.257727    2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0dd8bcac-f023-4c21-ad00-520136c355ac-image-registry-private-configuration\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:23:19.257874 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.257738    2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0dd8bcac-f023-4c21-ad00-520136c355ac-ca-trust-extracted\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:23:19.257874 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.257745    2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:23:19.257874 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:19.257754    2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0dd8bcac-f023-4c21-ad00-520136c355ac-registry-certificates\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:23:20.134128 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:20.134096    2570 generic.go:356] "Generic (PLEG): container finished" podID="9f681b31-f05e-4f17-8b0d-7aaac08ea039" containerID="00e17b4cb39b3d817ed1db84be7506b9e467bf541f238560010f59ed31d965f8" exitCode=0
Jul 07 02:23:20.134560 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:20.134165    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf" event={"ID":"9f681b31-f05e-4f17-8b0d-7aaac08ea039","Type":"ContainerDied","Data":"00e17b4cb39b3d817ed1db84be7506b9e467bf541f238560010f59ed31d965f8"}
Jul 07 02:23:20.134560 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:20.134248    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66bff4b87-w97mk"
Jul 07 02:23:20.134560 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:20.134443    2570 scope.go:117] "RemoveContainer" containerID="00e17b4cb39b3d817ed1db84be7506b9e467bf541f238560010f59ed31d965f8"
Jul 07 02:23:20.184421 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:20.184333    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66bff4b87-w97mk"]
Jul 07 02:23:20.189532 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:20.189503    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66bff4b87-w97mk"]
Jul 07 02:23:21.085897 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:21.085861    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dd8bcac-f023-4c21-ad00-520136c355ac" path="/var/lib/kubelet/pods/0dd8bcac-f023-4c21-ad00-520136c355ac/volumes"
Jul 07 02:23:21.139775 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:21.139723    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cpjkf" event={"ID":"9f681b31-f05e-4f17-8b0d-7aaac08ea039","Type":"ContainerStarted","Data":"e197cfcf428e100ad0a981ea2ccc1166b8abdc85db506e2925877cdf3f8ccd1a"}
Jul 07 02:23:21.468728 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:21.468619    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:21.468728 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:21.468684    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:36.203563 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:36.203522    2570 generic.go:356] "Generic (PLEG): container finished" podID="6b3022fc-1859-4338-bb06-cef488979501" containerID="802ac4a3713e13423eb3dd10157d72941c2da8f58b244660f5539fa0417e0ab2" exitCode=0
Jul 07 02:23:36.203911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:36.203586    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256" event={"ID":"6b3022fc-1859-4338-bb06-cef488979501","Type":"ContainerDied","Data":"802ac4a3713e13423eb3dd10157d72941c2da8f58b244660f5539fa0417e0ab2"}
Jul 07 02:23:36.203911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:36.203889    2570 scope.go:117] "RemoveContainer" containerID="802ac4a3713e13423eb3dd10157d72941c2da8f58b244660f5539fa0417e0ab2"
Jul 07 02:23:37.209112 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:37.209078    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-jz256" event={"ID":"6b3022fc-1859-4338-bb06-cef488979501","Type":"ContainerStarted","Data":"edd06ff9ca262a9eb4b564e73fa23ea302a5e93c5dad5bc3f34d569b504e993e"}
Jul 07 02:23:39.218547 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:39.218522    2570 generic.go:356] "Generic (PLEG): container finished" podID="f82c493c-8343-46b9-a50c-5dc1414d66f0" containerID="040a49416617f64cba7819da06afbba45a77cf2e9651c68ecdec2b33aea20f52" exitCode=0
Jul 07 02:23:39.218846 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:39.218579    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4" event={"ID":"f82c493c-8343-46b9-a50c-5dc1414d66f0","Type":"ContainerDied","Data":"040a49416617f64cba7819da06afbba45a77cf2e9651c68ecdec2b33aea20f52"}
Jul 07 02:23:39.218846 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:39.218841    2570 scope.go:117] "RemoveContainer" containerID="040a49416617f64cba7819da06afbba45a77cf2e9651c68ecdec2b33aea20f52"
Jul 07 02:23:40.225177 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:40.225143    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-gckv4" event={"ID":"f82c493c-8343-46b9-a50c-5dc1414d66f0","Type":"ContainerStarted","Data":"5fa55bb9f65f4114ab1899b997b8fc925c8f66e826cfa5bbe543e3ae515bf7be"}
Jul 07 02:23:41.473976 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:41.473947    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:41.477973 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:41.477950    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d85c655c5-kgzgq"
Jul 07 02:23:58.778366 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:58.778317    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:23:58.780702 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:58.780683    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e49d21df-0b01-4845-ad69-b2ad83b93187-metrics-certs\") pod \"network-metrics-daemon-wcsrt\" (UID: \"e49d21df-0b01-4845-ad69-b2ad83b93187\") " pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:23:58.785459 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:58.785441    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwgqf\""
Jul 07 02:23:58.793641 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:58.793620    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wcsrt"
Jul 07 02:23:58.924847 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:58.924684    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wcsrt"]
Jul 07 02:23:58.927311 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:23:58.927286    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49d21df_0b01_4845_ad69_b2ad83b93187.slice/crio-06a83a583b35878ec008a26c572696e39a8fef1ada489bfcb7bec5ff7499d9de WatchSource:0}: Error finding container 06a83a583b35878ec008a26c572696e39a8fef1ada489bfcb7bec5ff7499d9de: Status 404 returned error can't find the container with id 06a83a583b35878ec008a26c572696e39a8fef1ada489bfcb7bec5ff7499d9de
Jul 07 02:23:59.300279 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:23:59.300238    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wcsrt" event={"ID":"e49d21df-0b01-4845-ad69-b2ad83b93187","Type":"ContainerStarted","Data":"06a83a583b35878ec008a26c572696e39a8fef1ada489bfcb7bec5ff7499d9de"}
Jul 07 02:24:00.306096 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:00.306065    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wcsrt" event={"ID":"e49d21df-0b01-4845-ad69-b2ad83b93187","Type":"ContainerStarted","Data":"a01dc6cb3466834811973bd063e703b7721fe14156184e324e8bd6df0b9399a0"}
Jul 07 02:24:00.306525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:00.306114    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wcsrt" event={"ID":"e49d21df-0b01-4845-ad69-b2ad83b93187","Type":"ContainerStarted","Data":"b696bd93b5cdca24f3dbe60b4c01acffee7ee0585782f543705103a335fc098f"}
Jul 07 02:24:00.323068 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:00.323024    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wcsrt" podStartSLOduration=374.268237817 podStartE2EDuration="6m15.32300947s" totalImagesPullingTime="1.054771653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:24:00.321816256 +0000 UTC m=+375.729405935" watchObservedRunningTime="2026-07-07 02:24:00.32300947 +0000 UTC m=+375.730599159"
Jul 07 02:24:03.374985 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:03.374953    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:03.391104 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:03.391076    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:04.338455 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:04.338426    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:17.120750 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.120721    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:24:17.121213 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.120973    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="alertmanager" containerID="cri-o://981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e" gracePeriod=120
Jul 07 02:24:17.121213 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.121037    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy-metric" containerID="cri-o://5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e" gracePeriod=120
Jul 07 02:24:17.121213 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.121038    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy-web" containerID="cri-o://d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34" gracePeriod=120
Jul 07 02:24:17.121213 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.121049    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy" containerID="cri-o://9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f" gracePeriod=120
Jul 07 02:24:17.121213 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.121112    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="prom-label-proxy" containerID="cri-o://abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409" gracePeriod=120
Jul 07 02:24:17.121213 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.121094    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="config-reloader" containerID="cri-o://76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f" gracePeriod=120
Jul 07 02:24:17.379507 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.379435    2570 generic.go:356] "Generic (PLEG): container finished" podID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerID="abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409" exitCode=0
Jul 07 02:24:17.379507 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.379459    2570 generic.go:356] "Generic (PLEG): container finished" podID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerID="9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f" exitCode=0
Jul 07 02:24:17.379507 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.379467    2570 generic.go:356] "Generic (PLEG): container finished" podID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerID="76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f" exitCode=0
Jul 07 02:24:17.379507 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.379477    2570 generic.go:356] "Generic (PLEG): container finished" podID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerID="981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e" exitCode=0
Jul 07 02:24:17.379736 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.379507    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerDied","Data":"abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409"}
Jul 07 02:24:17.379736 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.379569    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerDied","Data":"9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f"}
Jul 07 02:24:17.379736 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.379597    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerDied","Data":"76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f"}
Jul 07 02:24:17.379736 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:17.379622    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerDied","Data":"981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e"}
Jul 07 02:24:18.371060 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.371038    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.386256 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.386229    2570 generic.go:356] "Generic (PLEG): container finished" podID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerID="5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e" exitCode=0
Jul 07 02:24:18.386256 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.386256    2570 generic.go:356] "Generic (PLEG): container finished" podID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerID="d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34" exitCode=0
Jul 07 02:24:18.386444 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.386312    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.386444 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.386314    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerDied","Data":"5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e"}
Jul 07 02:24:18.386444 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.386422    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerDied","Data":"d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34"}
Jul 07 02:24:18.386554 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.386468    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29ff97e9-80a6-4cb8-97f3-42b343309ecd","Type":"ContainerDied","Data":"512cbb01f030671fdc2d9b106575b0811ac963bdb90b49de63ac3b1508e9d4bf"}
Jul 07 02:24:18.386554 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.386490    2570 scope.go:117] "RemoveContainer" containerID="abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409"
Jul 07 02:24:18.396495 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.396480    2570 scope.go:117] "RemoveContainer" containerID="5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e"
Jul 07 02:24:18.405292 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.405244    2570 scope.go:117] "RemoveContainer" containerID="9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f"
Jul 07 02:24:18.414710 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.414688    2570 scope.go:117] "RemoveContainer" containerID="d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34"
Jul 07 02:24:18.422719 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.422697    2570 scope.go:117] "RemoveContainer" containerID="76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f"
Jul 07 02:24:18.429114 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429068    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-main-tls\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429114 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429111    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429259 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429127    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-web\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429259 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429153    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-metrics-client-ca\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429259 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429235    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-out\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429494 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429282    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-trusted-ca-bundle\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429494 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429317    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-main-db\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429494 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429359    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr6gl\" (UniqueName: \"kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-kube-api-access-xr6gl\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429494 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429388    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-tls-assets\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429494 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429450    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-volume\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429494 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429491    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429784 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429517    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-cluster-tls-config\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429784 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429558    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:24:18.429784 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429574    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-web-config\") pod \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\" (UID: \"29ff97e9-80a6-4cb8-97f3-42b343309ecd\") "
Jul 07 02:24:18.429784 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429742    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:24:18.430019 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.429945    2570 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-metrics-client-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.430101 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.430025    2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-main-db\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.431975 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.431644    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:24:18.431975 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.431934    2570 scope.go:117] "RemoveContainer" containerID="981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e"
Jul 07 02:24:18.432844 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.432428    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:18.434122 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.434076    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:18.434392 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.434254    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-volume" (OuterVolumeSpecName: "config-volume") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:18.434392 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.434314    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-out" (OuterVolumeSpecName: "config-out") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:24:18.434392 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.434323    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:18.435965 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.435927    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-kube-api-access-xr6gl" (OuterVolumeSpecName: "kube-api-access-xr6gl") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "kube-api-access-xr6gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:24:18.436093 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.435988    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:18.436093 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.436014    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:24:18.440567 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.440542    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:18.447155 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.447139    2570 scope.go:117] "RemoveContainer" containerID="5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e"
Jul 07 02:24:18.448095 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.448072    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-web-config" (OuterVolumeSpecName: "web-config") pod "29ff97e9-80a6-4cb8-97f3-42b343309ecd" (UID: "29ff97e9-80a6-4cb8-97f3-42b343309ecd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:18.454158 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.454140    2570 scope.go:117] "RemoveContainer" containerID="abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409"
Jul 07 02:24:18.454446 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:18.454425    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409\": container with ID starting with abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409 not found: ID does not exist" containerID="abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409"
Jul 07 02:24:18.454516 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.454456    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409"} err="failed to get container status \"abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409\": rpc error: code = NotFound desc = could not find container \"abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409\": container with ID starting with abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409 not found: ID does not exist"
Jul 07 02:24:18.454516 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.454498    2570 scope.go:117] "RemoveContainer" containerID="5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e"
Jul 07 02:24:18.454742 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:18.454726    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e\": container with ID starting with 5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e not found: ID does not exist" containerID="5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e"
Jul 07 02:24:18.454811 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.454749    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e"} err="failed to get container status \"5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e\": rpc error: code = NotFound desc = could not find container \"5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e\": container with ID starting with 5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e not found: ID does not exist"
Jul 07 02:24:18.454811 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.454767    2570 scope.go:117] "RemoveContainer" containerID="9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f"
Jul 07 02:24:18.455017 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:18.454995    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f\": container with ID starting with 9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f not found: ID does not exist" containerID="9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f"
Jul 07 02:24:18.455103 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.455021    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f"} err="failed to get container status \"9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f\": rpc error: code = NotFound desc = could not find container \"9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f\": container with ID starting with 9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f not found: ID does not exist"
Jul 07 02:24:18.455103 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.455034    2570 scope.go:117] "RemoveContainer" containerID="d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34"
Jul 07 02:24:18.455289 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:18.455270    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34\": container with ID starting with d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34 not found: ID does not exist" containerID="d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34"
Jul 07 02:24:18.455338 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.455293    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34"} err="failed to get container status \"d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34\": rpc error: code = NotFound desc = could not find container \"d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34\": container with ID starting with d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34 not found: ID does not exist"
Jul 07 02:24:18.455338 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.455308    2570 scope.go:117] "RemoveContainer" containerID="76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f"
Jul 07 02:24:18.455572 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:18.455557    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f\": container with ID starting with 76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f not found: ID does not exist" containerID="76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f"
Jul 07 02:24:18.455620 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.455575    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f"} err="failed to get container status \"76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f\": rpc error: code = NotFound desc = could not find container \"76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f\": container with ID starting with 76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f not found: ID does not exist"
Jul 07 02:24:18.455620 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.455587    2570 scope.go:117] "RemoveContainer" containerID="981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e"
Jul 07 02:24:18.455810 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:18.455791    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e\": container with ID starting with 981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e not found: ID does not exist" containerID="981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e"
Jul 07 02:24:18.455852 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.455815    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e"} err="failed to get container status \"981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e\": rpc error: code = NotFound desc = could not find container \"981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e\": container with ID starting with 981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e not found: ID does not exist"
Jul 07 02:24:18.455852 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.455830    2570 scope.go:117] "RemoveContainer" containerID="5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e"
Jul 07 02:24:18.456024 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:18.456008    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e\": container with ID starting with 5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e not found: ID does not exist" containerID="5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e"
Jul 07 02:24:18.456064 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456026    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e"} err="failed to get container status \"5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e\": rpc error: code = NotFound desc = could not find container \"5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e\": container with ID starting with 5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e not found: ID does not exist"
Jul 07 02:24:18.456064 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456038    2570 scope.go:117] "RemoveContainer" containerID="abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409"
Jul 07 02:24:18.456225 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456209    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409"} err="failed to get container status \"abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409\": rpc error: code = NotFound desc = could not find container \"abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409\": container with ID starting with abb290c9614898c0660fb33a1b05d06fac24dc8e27fc60bb83a6c9b59b979409 not found: ID does not exist"
Jul 07 02:24:18.456272 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456226    2570 scope.go:117] "RemoveContainer" containerID="5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e"
Jul 07 02:24:18.456410 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456394    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e"} err="failed to get container status \"5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e\": rpc error: code = NotFound desc = could not find container \"5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e\": container with ID starting with 5d2b8dc8a79c7d3cbd3340934804602119cbd999c9a53f3823e8c35379a5ac7e not found: ID does not exist"
Jul 07 02:24:18.456497 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456410    2570 scope.go:117] "RemoveContainer" containerID="9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f"
Jul 07 02:24:18.456618 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456600    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f"} err="failed to get container status \"9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f\": rpc error: code = NotFound desc = could not find container \"9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f\": container with ID starting with 9305d2698d3358abcb41574e209f4e30cb93354688609f594611e8a93825f66f not found: ID does not exist"
Jul 07 02:24:18.456659 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456618    2570 scope.go:117] "RemoveContainer" containerID="d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34"
Jul 07 02:24:18.456786 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456771    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34"} err="failed to get container status \"d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34\": rpc error: code = NotFound desc = could not find container \"d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34\": container with ID starting with d7b3da1c1e822aba3f3099cf07d5f2c1a8a59eaf6a9f191cd2b89f733d40db34 not found: ID does not exist"
Jul 07 02:24:18.456837 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456788    2570 scope.go:117] "RemoveContainer" containerID="76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f"
Jul 07 02:24:18.456961 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456939    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f"} err="failed to get container status \"76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f\": rpc error: code = NotFound desc = could not find container \"76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f\": container with ID starting with 76a39b70ec97455ece6936e7c2545226abdf74ab0587f971a8be97f9993cbb6f not found: ID does not exist"
Jul 07 02:24:18.457005 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.456961    2570 scope.go:117] "RemoveContainer" containerID="981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e"
Jul 07 02:24:18.457119 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.457103    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e"} err="failed to get container status \"981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e\": rpc error: code = NotFound desc = could not find container \"981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e\": container with ID starting with 981339d06f89958d66e39ebaf0da0ff7cbd1338db237a9cfc906a4432e06649e not found: ID does not exist"
Jul 07 02:24:18.457160 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.457118    2570 scope.go:117] "RemoveContainer" containerID="5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e"
Jul 07 02:24:18.457312 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.457296    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e"} err="failed to get container status \"5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e\": rpc error: code = NotFound desc = could not find container \"5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e\": container with ID starting with 5dd13bc328a6443f7517d408bd706f806324ccc3bfba572a4869f7a7ff0bea3e not found: ID does not exist"
Jul 07 02:24:18.531383 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.531332    2570 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-volume\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.531383 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.531381    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.531546 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.531396    2570 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-cluster-tls-config\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.531546 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.531409    2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-web-config\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.531546 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.531421    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-main-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.531546 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.531434    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.531546 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.531447    2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29ff97e9-80a6-4cb8-97f3-42b343309ecd-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.531546 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.531460    2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29ff97e9-80a6-4cb8-97f3-42b343309ecd-config-out\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.531546 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.531472    2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29ff97e9-80a6-4cb8-97f3-42b343309ecd-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.531546 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.531486    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr6gl\" (UniqueName: \"kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-kube-api-access-xr6gl\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.531546 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.531500    2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29ff97e9-80a6-4cb8-97f3-42b343309ecd-tls-assets\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:18.720184 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.720156    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:24:18.726130 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.722714    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:24:18.756979 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.756954    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:24:18.757251 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757240    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy-web"
Jul 07 02:24:18.757297 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757253    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy-web"
Jul 07 02:24:18.757297 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757265    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="alertmanager"
Jul 07 02:24:18.757297 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757270    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="alertmanager"
Jul 07 02:24:18.757297 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757279    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="init-config-reloader"
Jul 07 02:24:18.757297 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757284    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="init-config-reloader"
Jul 07 02:24:18.757297 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757289    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="prom-label-proxy"
Jul 07 02:24:18.757297 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757294    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="prom-label-proxy"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757307    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="config-reloader"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757313    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="config-reloader"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757318    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy-metric"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757323    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy-metric"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757329    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757333    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757356    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd8bcac-f023-4c21-ad00-520136c355ac" containerName="registry"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757361    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd8bcac-f023-4c21-ad00-520136c355ac" containerName="registry"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757412    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy-metric"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757421    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="prom-label-proxy"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757427    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757432    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="config-reloader"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757437    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="kube-rbac-proxy-web"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757445    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" containerName="alertmanager"
Jul 07 02:24:18.757534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.757451    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd8bcac-f023-4c21-ad00-520136c355ac" containerName="registry"
Jul 07 02:24:18.760967 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.760952    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.763737 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.763718    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 07 02:24:18.763852 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.763743    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 07 02:24:18.763852 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.763743    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 07 02:24:18.764059 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.764042    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5cp5g\""
Jul 07 02:24:18.764145 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.764127    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 07 02:24:18.764305 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.764291    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 07 02:24:18.764375 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.764339    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 07 02:24:18.764689 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.764671    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 07 02:24:18.765060 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.765016    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 07 02:24:18.770381 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.770361    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 07 02:24:18.779576 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.779558    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:24:18.833625 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.833601    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b570cce1-9eea-4a4a-8c93-54afa0e65736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.833736 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.833631    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.833736 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.833651    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5x6\" (UniqueName: \"kubernetes.io/projected/b570cce1-9eea-4a4a-8c93-54afa0e65736-kube-api-access-hv5x6\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.833736 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.833676    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-web-config\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.833736 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.833725    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b570cce1-9eea-4a4a-8c93-54afa0e65736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.833926 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.833765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b570cce1-9eea-4a4a-8c93-54afa0e65736-config-out\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.833926 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.833800    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.833926 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.833867    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b570cce1-9eea-4a4a-8c93-54afa0e65736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.833926 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.833905    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.834110 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.833945    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b570cce1-9eea-4a4a-8c93-54afa0e65736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.834110 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.833972    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-config-volume\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.834110 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.834001    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.834110 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.834050    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.934579 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934553    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.934680 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934585    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b570cce1-9eea-4a4a-8c93-54afa0e65736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.934680 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934602    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-config-volume\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.934680 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934630    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.934680 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934667    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.934864 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934703    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b570cce1-9eea-4a4a-8c93-54afa0e65736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.934864 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934728    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.934864 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934824    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5x6\" (UniqueName: \"kubernetes.io/projected/b570cce1-9eea-4a4a-8c93-54afa0e65736-kube-api-access-hv5x6\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.935013 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934882    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-web-config\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.935013 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934911    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b570cce1-9eea-4a4a-8c93-54afa0e65736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.935013 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934952    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b570cce1-9eea-4a4a-8c93-54afa0e65736-config-out\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.935013 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.934994    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.935219 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.935032    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b570cce1-9eea-4a4a-8c93-54afa0e65736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.935219 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.935178    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b570cce1-9eea-4a4a-8c93-54afa0e65736-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.935867 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.935591    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b570cce1-9eea-4a4a-8c93-54afa0e65736-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.936365 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.936320    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b570cce1-9eea-4a4a-8c93-54afa0e65736-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.937894 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.937869    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.938103 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.938076    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b570cce1-9eea-4a4a-8c93-54afa0e65736-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.938234 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.938217    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-config-volume\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.938303 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.938247    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.938303 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.938268    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.938303 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.938267    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b570cce1-9eea-4a4a-8c93-54afa0e65736-config-out\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.938640 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.938619    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.939133 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.939110    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-web-config\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.940194 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.940171    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b570cce1-9eea-4a4a-8c93-54afa0e65736-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:18.944445 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:18.944427    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5x6\" (UniqueName: \"kubernetes.io/projected/b570cce1-9eea-4a4a-8c93-54afa0e65736-kube-api-access-hv5x6\") pod \"alertmanager-main-0\" (UID: \"b570cce1-9eea-4a4a-8c93-54afa0e65736\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:19.070142 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:19.070118    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 07 02:24:19.088117 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:19.088094    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29ff97e9-80a6-4cb8-97f3-42b343309ecd" path="/var/lib/kubelet/pods/29ff97e9-80a6-4cb8-97f3-42b343309ecd/volumes"
Jul 07 02:24:19.207489 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:19.207429    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 07 02:24:19.209729 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:24:19.209692    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb570cce1_9eea_4a4a_8c93_54afa0e65736.slice/crio-25b9831e8576eef2189a9f783a650e71c8e3a1394b34ebb8da651299b71d3a68 WatchSource:0}: Error finding container 25b9831e8576eef2189a9f783a650e71c8e3a1394b34ebb8da651299b71d3a68: Status 404 returned error can't find the container with id 25b9831e8576eef2189a9f783a650e71c8e3a1394b34ebb8da651299b71d3a68
Jul 07 02:24:19.391547 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:19.391492    2570 generic.go:356] "Generic (PLEG): container finished" podID="b570cce1-9eea-4a4a-8c93-54afa0e65736" containerID="6a2ebcc6a133ae3eb5e3e1d3f69d32413e1de1d5a7fe617479d9b7b7dfd463b2" exitCode=0
Jul 07 02:24:19.391909 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:19.391584    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b570cce1-9eea-4a4a-8c93-54afa0e65736","Type":"ContainerDied","Data":"6a2ebcc6a133ae3eb5e3e1d3f69d32413e1de1d5a7fe617479d9b7b7dfd463b2"}
Jul 07 02:24:19.391909 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:19.391658    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b570cce1-9eea-4a4a-8c93-54afa0e65736","Type":"ContainerStarted","Data":"25b9831e8576eef2189a9f783a650e71c8e3a1394b34ebb8da651299b71d3a68"}
Jul 07 02:24:20.400820 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:20.400789    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b570cce1-9eea-4a4a-8c93-54afa0e65736","Type":"ContainerStarted","Data":"e3fb86cf3b469c9384687b95ce64a04e5f72cc2bc1336553780f2524a05ead8b"}
Jul 07 02:24:20.401161 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:20.400851    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b570cce1-9eea-4a4a-8c93-54afa0e65736","Type":"ContainerStarted","Data":"862f7b1a67ad85a80d5ce35ad7b326b4962666ca5a02f46b95205642859a72ab"}
Jul 07 02:24:20.401161 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:20.400876    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b570cce1-9eea-4a4a-8c93-54afa0e65736","Type":"ContainerStarted","Data":"05f25685bb6b88591e53c16dcccdc67e45545cb2784b9a32307b37435394a171"}
Jul 07 02:24:20.401161 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:20.400900    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b570cce1-9eea-4a4a-8c93-54afa0e65736","Type":"ContainerStarted","Data":"cc268a750181b5e60ba92b85857e4e58e46a8c074c81c65f80a29c86a2364efd"}
Jul 07 02:24:20.401161 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:20.400923    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b570cce1-9eea-4a4a-8c93-54afa0e65736","Type":"ContainerStarted","Data":"ae05705fb90dc4d4cad0e672defa81b595d69c9bcdaa0d98b7547ac13603e644"}
Jul 07 02:24:20.401161 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:20.400945    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b570cce1-9eea-4a4a-8c93-54afa0e65736","Type":"ContainerStarted","Data":"19b09f414a651c9f02d6babeb35fb479ea94d14f92e9a89f89168c44b8a69b16"}
Jul 07 02:24:21.449206 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.449156    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.432389472 podStartE2EDuration="3.449140985s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.751513ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:24:20.430633575 +0000 UTC m=+395.838223267" watchObservedRunningTime="2026-07-07 02:24:21.449140985 +0000 UTC m=+396.856730672"
Jul 07 02:24:21.449674 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.449501    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:24:21.449763 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.449740    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="prometheus" containerID="cri-o://a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc" gracePeriod=600
Jul 07 02:24:21.449813 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.449775    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="thanos-sidecar" containerID="cri-o://061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1" gracePeriod=600
Jul 07 02:24:21.449867 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.449784    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy" containerID="cri-o://126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5" gracePeriod=600
Jul 07 02:24:21.449867 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.449805    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="config-reloader" containerID="cri-o://e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c" gracePeriod=600
Jul 07 02:24:21.449867 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.449822    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy-web" containerID="cri-o://36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa" gracePeriod=600
Jul 07 02:24:21.450009 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.449906    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e" gracePeriod=600
Jul 07 02:24:21.717454 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.717430    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:21.755502 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755477    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.755626 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755512    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-kube-rbac-proxy\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.755626 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755538    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-2\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.755626 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755576    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-metrics-client-ca\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.755626 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755613    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-serving-certs-ca-bundle\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.755842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755659    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-tls\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.755842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755684    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-grpc-tls\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.755842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755710    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-0\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.755842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755735    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-config-out\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.755842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755759    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-web-config\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.755842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755785    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.755842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755832    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-tls-assets\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.756149 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755857    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb9qf\" (UniqueName: \"kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-kube-api-access-pb9qf\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.756149 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755891    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-db\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.756149 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.755941    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-thanos-prometheus-http-client-file\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.756149 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.756007    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-kubelet-serving-ca-bundle\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.756149 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.756032    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-config\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.756149 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.756078    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-metrics-client-certs\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.756149 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.756108    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-trusted-ca-bundle\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.756149 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.756137    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-1\") pod \"51384e2b-4f02-42f0-a4b0-b99a21730f94\" (UID: \"51384e2b-4f02-42f0-a4b0-b99a21730f94\") "
Jul 07 02:24:21.756558 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.756406    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:24:21.756645 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.756622    2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.757306 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.756909    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:24:21.757306 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.756964    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:24:21.757306 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.757064    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:24:21.757629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.757578    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:24:21.758442 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.758414    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:24:21.758985 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.758957    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:24:21.760537 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.760512    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:24:21.761535 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.761510    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:21.761681 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.761539    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:21.762710 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.762599    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:21.762710 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.762676    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:21.762710 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.762692    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-config-out" (OuterVolumeSpecName: "config-out") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:24:21.763046 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.763022    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-config" (OuterVolumeSpecName: "config") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:21.763160 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.763136    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:24:21.763223 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.763157    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-kube-api-access-pb9qf" (OuterVolumeSpecName: "kube-api-access-pb9qf") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "kube-api-access-pb9qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:24:21.763223 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.763173    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:21.763330 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.763273    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:21.764037 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.764013    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:21.774020 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.773999    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-web-config" (OuterVolumeSpecName: "web-config") pod "51384e2b-4f02-42f0-a4b0-b99a21730f94" (UID: "51384e2b-4f02-42f0-a4b0-b99a21730f94"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:24:21.857579 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857555    2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857579 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857578    2570 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-grpc-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857587    2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857597    2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-config-out\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857605    2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-web-config\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857614    2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857622    2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-tls-assets\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857631    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb9qf\" (UniqueName: \"kubernetes.io/projected/51384e2b-4f02-42f0-a4b0-b99a21730f94-kube-api-access-pb9qf\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857639    2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-db\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857647    2570 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857656    2570 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857667    2570 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-config\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857674    2570 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-metrics-client-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.857679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857684    2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.858013 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857692    2570 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.858013 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857701    2570 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.858013 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857716    2570 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51384e2b-4f02-42f0-a4b0-b99a21730f94-secret-kube-rbac-proxy\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.858013 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857726    2570 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-metrics-client-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:21.858013 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:21.857734    2570 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51384e2b-4f02-42f0-a4b0-b99a21730f94-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:24:22.413525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413492    2570 generic.go:356] "Generic (PLEG): container finished" podID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerID="7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e" exitCode=0
Jul 07 02:24:22.413525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413520    2570 generic.go:356] "Generic (PLEG): container finished" podID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerID="126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5" exitCode=0
Jul 07 02:24:22.413525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413526    2570 generic.go:356] "Generic (PLEG): container finished" podID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerID="36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa" exitCode=0
Jul 07 02:24:22.413765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413538    2570 generic.go:356] "Generic (PLEG): container finished" podID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerID="061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1" exitCode=0
Jul 07 02:24:22.413765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413544    2570 generic.go:356] "Generic (PLEG): container finished" podID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerID="e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c" exitCode=0
Jul 07 02:24:22.413765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413549    2570 generic.go:356] "Generic (PLEG): container finished" podID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerID="a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc" exitCode=0
Jul 07 02:24:22.413765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413571    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.413765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413570    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerDied","Data":"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"}
Jul 07 02:24:22.413765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413650    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerDied","Data":"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"}
Jul 07 02:24:22.413765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413683    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerDied","Data":"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"}
Jul 07 02:24:22.413765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413706    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerDied","Data":"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"}
Jul 07 02:24:22.413765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413724    2570 scope.go:117] "RemoveContainer" containerID="7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"
Jul 07 02:24:22.413765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413743    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerDied","Data":"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"}
Jul 07 02:24:22.413765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413768    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerDied","Data":"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"}
Jul 07 02:24:22.414149 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.413803    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"51384e2b-4f02-42f0-a4b0-b99a21730f94","Type":"ContainerDied","Data":"d971b861cece61f110fd34eeea224f18d20ae3d3bf619a1af366d43a5b3dc4bd"}
Jul 07 02:24:22.422511 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.422476    2570 scope.go:117] "RemoveContainer" containerID="126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"
Jul 07 02:24:22.429755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.429738    2570 scope.go:117] "RemoveContainer" containerID="36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"
Jul 07 02:24:22.436584 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.436567    2570 scope.go:117] "RemoveContainer" containerID="061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"
Jul 07 02:24:22.438750 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.438728    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:24:22.443650 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.443629    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:24:22.446220 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.446202    2570 scope.go:117] "RemoveContainer" containerID="e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"
Jul 07 02:24:22.453069 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.452913    2570 scope.go:117] "RemoveContainer" containerID="a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"
Jul 07 02:24:22.460233 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.460209    2570 scope.go:117] "RemoveContainer" containerID="6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"
Jul 07 02:24:22.468375 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468340    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:24:22.468698 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468685    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="thanos-sidecar"
Jul 07 02:24:22.468758 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468701    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="thanos-sidecar"
Jul 07 02:24:22.468758 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468712    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="config-reloader"
Jul 07 02:24:22.468758 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468718    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="config-reloader"
Jul 07 02:24:22.468758 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468735    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="init-config-reloader"
Jul 07 02:24:22.468758 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468741    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="init-config-reloader"
Jul 07 02:24:22.468758 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468749    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="prometheus"
Jul 07 02:24:22.468758 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468754    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="prometheus"
Jul 07 02:24:22.468758 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468761    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy-web"
Jul 07 02:24:22.469032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468766    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy-web"
Jul 07 02:24:22.469032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468773    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy"
Jul 07 02:24:22.469032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468777    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy"
Jul 07 02:24:22.469032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468784    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy-thanos"
Jul 07 02:24:22.469032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468789    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy-thanos"
Jul 07 02:24:22.469032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468834    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="thanos-sidecar"
Jul 07 02:24:22.469032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468846    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="prometheus"
Jul 07 02:24:22.469032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468854    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy"
Jul 07 02:24:22.469032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468860    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy-thanos"
Jul 07 02:24:22.469032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468866    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="config-reloader"
Jul 07 02:24:22.469032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.468873    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" containerName="kube-rbac-proxy-web"
Jul 07 02:24:22.472060 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.472044    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.474621 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.474591    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 07 02:24:22.474763 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.474747    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 07 02:24:22.474907 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.474886    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 07 02:24:22.475053 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.474975    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 07 02:24:22.475117 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.475101    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dkhanorfma7ch\""
Jul 07 02:24:22.475170 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.475114    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 07 02:24:22.475268 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.475178    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 07 02:24:22.475423 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.475404    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 07 02:24:22.475423 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.475412    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kv6z7\""
Jul 07 02:24:22.475579 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.475462    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 07 02:24:22.475706 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.475639    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 07 02:24:22.475706 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.475655    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 07 02:24:22.475868 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.475745    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 07 02:24:22.475868 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.475799    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 07 02:24:22.479400 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.479380    2570 scope.go:117] "RemoveContainer" containerID="7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"
Jul 07 02:24:22.479750 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:22.479725    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": container with ID starting with 7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e not found: ID does not exist" containerID="7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"
Jul 07 02:24:22.485405 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.479762    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"} err="failed to get container status \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": rpc error: code = NotFound desc = could not find container \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": container with ID starting with 7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e not found: ID does not exist"
Jul 07 02:24:22.485405 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.480464    2570 scope.go:117] "RemoveContainer" containerID="126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"
Jul 07 02:24:22.485405 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.482053    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 07 02:24:22.486498 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:22.486228    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": container with ID starting with 126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5 not found: ID does not exist" containerID="126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"
Jul 07 02:24:22.486498 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.486298    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"} err="failed to get container status \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": rpc error: code = NotFound desc = could not find container \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": container with ID starting with 126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5 not found: ID does not exist"
Jul 07 02:24:22.486498 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.486320    2570 scope.go:117] "RemoveContainer" containerID="36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"
Jul 07 02:24:22.487686 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:22.487070    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": container with ID starting with 36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa not found: ID does not exist" containerID="36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"
Jul 07 02:24:22.487686 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.487102    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"} err="failed to get container status \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": rpc error: code = NotFound desc = could not find container \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": container with ID starting with 36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa not found: ID does not exist"
Jul 07 02:24:22.487686 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.487131    2570 scope.go:117] "RemoveContainer" containerID="061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"
Jul 07 02:24:22.487686 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.487474    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 07 02:24:22.487686 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:22.487558    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": container with ID starting with 061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1 not found: ID does not exist" containerID="061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"
Jul 07 02:24:22.487686 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.487614    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"} err="failed to get container status \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": rpc error: code = NotFound desc = could not find container \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": container with ID starting with 061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1 not found: ID does not exist"
Jul 07 02:24:22.487686 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.487637    2570 scope.go:117] "RemoveContainer" containerID="e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"
Jul 07 02:24:22.488921 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:22.488885    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": container with ID starting with e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c not found: ID does not exist" containerID="e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"
Jul 07 02:24:22.489166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.488930    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"} err="failed to get container status \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": rpc error: code = NotFound desc = could not find container \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": container with ID starting with e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c not found: ID does not exist"
Jul 07 02:24:22.489467 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.489177    2570 scope.go:117] "RemoveContainer" containerID="a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"
Jul 07 02:24:22.489753 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:22.489733    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": container with ID starting with a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc not found: ID does not exist" containerID="a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"
Jul 07 02:24:22.489845 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.489760    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"} err="failed to get container status \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": rpc error: code = NotFound desc = could not find container \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": container with ID starting with a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc not found: ID does not exist"
Jul 07 02:24:22.489845 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.489775    2570 scope.go:117] "RemoveContainer" containerID="6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"
Jul 07 02:24:22.490040 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:24:22.490025    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": container with ID starting with 6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773 not found: ID does not exist" containerID="6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"
Jul 07 02:24:22.490105 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.490045    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"} err="failed to get container status \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": rpc error: code = NotFound desc = could not find container \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": container with ID starting with 6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773 not found: ID does not exist"
Jul 07 02:24:22.490105 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.490057    2570 scope.go:117] "RemoveContainer" containerID="7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"
Jul 07 02:24:22.490271 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.490256    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"} err="failed to get container status \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": rpc error: code = NotFound desc = could not find container \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": container with ID starting with 7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e not found: ID does not exist"
Jul 07 02:24:22.490331 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.490271    2570 scope.go:117] "RemoveContainer" containerID="126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"
Jul 07 02:24:22.490573 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.490549    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"} err="failed to get container status \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": rpc error: code = NotFound desc = could not find container \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": container with ID starting with 126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5 not found: ID does not exist"
Jul 07 02:24:22.490652 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.490575    2570 scope.go:117] "RemoveContainer" containerID="36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"
Jul 07 02:24:22.490824 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.490805    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"} err="failed to get container status \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": rpc error: code = NotFound desc = could not find container \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": container with ID starting with 36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa not found: ID does not exist"
Jul 07 02:24:22.490896 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.490824    2570 scope.go:117] "RemoveContainer" containerID="061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"
Jul 07 02:24:22.490896 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.490830    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:24:22.491072 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.491049    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"} err="failed to get container status \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": rpc error: code = NotFound desc = could not find container \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": container with ID starting with 061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1 not found: ID does not exist"
Jul 07 02:24:22.491072 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.491070    2570 scope.go:117] "RemoveContainer" containerID="e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"
Jul 07 02:24:22.491330 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.491307    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"} err="failed to get container status \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": rpc error: code = NotFound desc = could not find container \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": container with ID starting with e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c not found: ID does not exist"
Jul 07 02:24:22.491429 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.491332    2570 scope.go:117] "RemoveContainer" containerID="a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"
Jul 07 02:24:22.491677 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.491644    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"} err="failed to get container status \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": rpc error: code = NotFound desc = could not find container \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": container with ID starting with a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc not found: ID does not exist"
Jul 07 02:24:22.491677 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.491666    2570 scope.go:117] "RemoveContainer" containerID="6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"
Jul 07 02:24:22.491948 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.491929    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"} err="failed to get container status \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": rpc error: code = NotFound desc = could not find container \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": container with ID starting with 6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773 not found: ID does not exist"
Jul 07 02:24:22.492029 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.491950    2570 scope.go:117] "RemoveContainer" containerID="7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"
Jul 07 02:24:22.492210 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.492193    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"} err="failed to get container status \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": rpc error: code = NotFound desc = could not find container \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": container with ID starting with 7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e not found: ID does not exist"
Jul 07 02:24:22.492271 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.492211    2570 scope.go:117] "RemoveContainer" containerID="126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"
Jul 07 02:24:22.492432 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.492416    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"} err="failed to get container status \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": rpc error: code = NotFound desc = could not find container \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": container with ID starting with 126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5 not found: ID does not exist"
Jul 07 02:24:22.492492 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.492432    2570 scope.go:117] "RemoveContainer" containerID="36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"
Jul 07 02:24:22.492638 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.492620    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"} err="failed to get container status \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": rpc error: code = NotFound desc = could not find container \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": container with ID starting with 36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa not found: ID does not exist"
Jul 07 02:24:22.492638 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.492636    2570 scope.go:117] "RemoveContainer" containerID="061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"
Jul 07 02:24:22.492814 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.492796    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"} err="failed to get container status \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": rpc error: code = NotFound desc = could not find container \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": container with ID starting with 061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1 not found: ID does not exist"
Jul 07 02:24:22.492814 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.492813    2570 scope.go:117] "RemoveContainer" containerID="e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"
Jul 07 02:24:22.493008 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.492981    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"} err="failed to get container status \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": rpc error: code = NotFound desc = could not find container \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": container with ID starting with e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c not found: ID does not exist"
Jul 07 02:24:22.493008 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.493007    2570 scope.go:117] "RemoveContainer" containerID="a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"
Jul 07 02:24:22.493269 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.493250    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"} err="failed to get container status \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": rpc error: code = NotFound desc = could not find container \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": container with ID starting with a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc not found: ID does not exist"
Jul 07 02:24:22.493317 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.493271    2570 scope.go:117] "RemoveContainer" containerID="6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"
Jul 07 02:24:22.493530 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.493511    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"} err="failed to get container status \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": rpc error: code = NotFound desc = could not find container \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": container with ID starting with 6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773 not found: ID does not exist"
Jul 07 02:24:22.493576 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.493532    2570 scope.go:117] "RemoveContainer" containerID="7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"
Jul 07 02:24:22.493752 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.493732    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"} err="failed to get container status \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": rpc error: code = NotFound desc = could not find container \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": container with ID starting with 7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e not found: ID does not exist"
Jul 07 02:24:22.493794 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.493753    2570 scope.go:117] "RemoveContainer" containerID="126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"
Jul 07 02:24:22.493999 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.493977    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"} err="failed to get container status \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": rpc error: code = NotFound desc = could not find container \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": container with ID starting with 126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5 not found: ID does not exist"
Jul 07 02:24:22.494061 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.494001    2570 scope.go:117] "RemoveContainer" containerID="36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"
Jul 07 02:24:22.494246 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.494228    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"} err="failed to get container status \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": rpc error: code = NotFound desc = could not find container \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": container with ID starting with 36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa not found: ID does not exist"
Jul 07 02:24:22.494303 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.494247    2570 scope.go:117] "RemoveContainer" containerID="061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"
Jul 07 02:24:22.494491 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.494475    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"} err="failed to get container status \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": rpc error: code = NotFound desc = could not find container \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": container with ID starting with 061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1 not found: ID does not exist"
Jul 07 02:24:22.494539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.494493    2570 scope.go:117] "RemoveContainer" containerID="e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"
Jul 07 02:24:22.494718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.494702    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"} err="failed to get container status \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": rpc error: code = NotFound desc = could not find container \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": container with ID starting with e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c not found: ID does not exist"
Jul 07 02:24:22.494789 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.494720    2570 scope.go:117] "RemoveContainer" containerID="a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"
Jul 07 02:24:22.494942 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.494919    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"} err="failed to get container status \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": rpc error: code = NotFound desc = could not find container \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": container with ID starting with a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc not found: ID does not exist"
Jul 07 02:24:22.494980 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.494945    2570 scope.go:117] "RemoveContainer" containerID="6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"
Jul 07 02:24:22.495179 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.495164    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"} err="failed to get container status \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": rpc error: code = NotFound desc = could not find container \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": container with ID starting with 6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773 not found: ID does not exist"
Jul 07 02:24:22.495230 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.495179    2570 scope.go:117] "RemoveContainer" containerID="7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"
Jul 07 02:24:22.495397 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.495382    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"} err="failed to get container status \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": rpc error: code = NotFound desc = could not find container \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": container with ID starting with 7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e not found: ID does not exist"
Jul 07 02:24:22.495438 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.495397    2570 scope.go:117] "RemoveContainer" containerID="126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"
Jul 07 02:24:22.495630 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.495604    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"} err="failed to get container status \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": rpc error: code = NotFound desc = could not find container \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": container with ID starting with 126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5 not found: ID does not exist"
Jul 07 02:24:22.495671 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.495634    2570 scope.go:117] "RemoveContainer" containerID="36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"
Jul 07 02:24:22.495851 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.495836    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"} err="failed to get container status \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": rpc error: code = NotFound desc = could not find container \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": container with ID starting with 36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa not found: ID does not exist"
Jul 07 02:24:22.495896 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.495852    2570 scope.go:117] "RemoveContainer" containerID="061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"
Jul 07 02:24:22.496064 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.496049    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"} err="failed to get container status \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": rpc error: code = NotFound desc = could not find container \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": container with ID starting with 061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1 not found: ID does not exist"
Jul 07 02:24:22.496108 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.496065    2570 scope.go:117] "RemoveContainer" containerID="e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"
Jul 07 02:24:22.496250 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.496236    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"} err="failed to get container status \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": rpc error: code = NotFound desc = could not find container \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": container with ID starting with e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c not found: ID does not exist"
Jul 07 02:24:22.496290 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.496250    2570 scope.go:117] "RemoveContainer" containerID="a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"
Jul 07 02:24:22.496508 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.496486    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"} err="failed to get container status \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": rpc error: code = NotFound desc = could not find container \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": container with ID starting with a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc not found: ID does not exist"
Jul 07 02:24:22.496570 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.496511    2570 scope.go:117] "RemoveContainer" containerID="6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"
Jul 07 02:24:22.496740 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.496723    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"} err="failed to get container status \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": rpc error: code = NotFound desc = could not find container \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": container with ID starting with 6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773 not found: ID does not exist"
Jul 07 02:24:22.496781 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.496741    2570 scope.go:117] "RemoveContainer" containerID="7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"
Jul 07 02:24:22.496973 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.496955    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e"} err="failed to get container status \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": rpc error: code = NotFound desc = could not find container \"7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e\": container with ID starting with 7a3e8c8aa0fb1d2de7853e943095332a7077a72e26c40acb01d7d6e5da2c577e not found: ID does not exist"
Jul 07 02:24:22.497025 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.496973    2570 scope.go:117] "RemoveContainer" containerID="126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"
Jul 07 02:24:22.497175 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.497159    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5"} err="failed to get container status \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": rpc error: code = NotFound desc = could not find container \"126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5\": container with ID starting with 126d27eddbc9db56e6a176ae3f2b8f10f1a7466933cabe32f73760712c6744d5 not found: ID does not exist"
Jul 07 02:24:22.497216 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.497175    2570 scope.go:117] "RemoveContainer" containerID="36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"
Jul 07 02:24:22.497390 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.497374    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa"} err="failed to get container status \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": rpc error: code = NotFound desc = could not find container \"36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa\": container with ID starting with 36f91940cfb638b8d187bfe19977c5733817405be0d5987e7de6abae975aa4fa not found: ID does not exist"
Jul 07 02:24:22.497440 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.497389    2570 scope.go:117] "RemoveContainer" containerID="061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"
Jul 07 02:24:22.497621 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.497592    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1"} err="failed to get container status \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": rpc error: code = NotFound desc = could not find container \"061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1\": container with ID starting with 061629ed6bdbf8f3947ee811cb2157662fb72c0f8130e0a28d95c935e3cce4a1 not found: ID does not exist"
Jul 07 02:24:22.497667 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.497624    2570 scope.go:117] "RemoveContainer" containerID="e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"
Jul 07 02:24:22.497882 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.497866    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c"} err="failed to get container status \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": rpc error: code = NotFound desc = could not find container \"e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c\": container with ID starting with e89d0dc8931335d38de08b72c7b449ec7bdfc480209c50dcd9ac984d1443476c not found: ID does not exist"
Jul 07 02:24:22.497982 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.497881    2570 scope.go:117] "RemoveContainer" containerID="a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"
Jul 07 02:24:22.498082 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.498068    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc"} err="failed to get container status \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": rpc error: code = NotFound desc = could not find container \"a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc\": container with ID starting with a4b103e1654bfb8602c18839dae9910924fb824320ff0a75f2bd531b7b4334fc not found: ID does not exist"
Jul 07 02:24:22.498118 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.498084    2570 scope.go:117] "RemoveContainer" containerID="6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"
Jul 07 02:24:22.498273 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.498250    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773"} err="failed to get container status \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": rpc error: code = NotFound desc = could not find container \"6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773\": container with ID starting with 6897e0ecb1eca0831c8a49d33c917d69c1211eb3ef24e60585822299ab721773 not found: ID does not exist"
Jul 07 02:24:22.564178 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564149    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564269 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564184    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-web-config\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564269 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564203    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564269 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564234    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564269 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564438 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564274    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564438 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564296    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564438 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564325    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564438 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564370    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/661c1aad-7c80-40b1-9938-3564b19025da-config-out\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564438 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564405    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564438 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564432    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/661c1aad-7c80-40b1-9938-3564b19025da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564631 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564448    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564631 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564488    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564631 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564517    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564631 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564535    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564631 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564551    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-config\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564631 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564567    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564631 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564585    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llg4\" (UniqueName: \"kubernetes.io/projected/661c1aad-7c80-40b1-9938-3564b19025da-kube-api-access-7llg4\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564631 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564610    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.564631 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.564625    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.665599 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.665537    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.665599 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.665577    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.665705 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.665604    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.665705 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.665632    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.665705 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.665654    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-config\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.665705 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.665682    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.665886 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.665707    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llg4\" (UniqueName: \"kubernetes.io/projected/661c1aad-7c80-40b1-9938-3564b19025da-kube-api-access-7llg4\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.665886 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.665770    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.666911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666214    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.666911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666241    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.666911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666747    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.666911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666247    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.666911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666807    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-web-config\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.666911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666837    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.666911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666872    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.666911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666903    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.667362 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666930    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.667362 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666969    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.667362 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666994    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.667362 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.666995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.667362 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.667060    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/661c1aad-7c80-40b1-9938-3564b19025da-config-out\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.667362 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.667104    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.667362 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.667153    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/661c1aad-7c80-40b1-9938-3564b19025da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.669376 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.668849    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.669376 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.668866    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.669376 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.668986    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.669376 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.669048    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-config\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.669976 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.669701    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.669976 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.669808    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.669976 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.669813    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.670583 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.670511    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.670583 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.670526    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/661c1aad-7c80-40b1-9938-3564b19025da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.670832 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.670803    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.671198 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.671174    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/661c1aad-7c80-40b1-9938-3564b19025da-config-out\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.671391 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.671370    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.671761 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.671726    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-web-config\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.671845 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.671765    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/661c1aad-7c80-40b1-9938-3564b19025da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.672394 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.672373    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.672896 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.672875    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/661c1aad-7c80-40b1-9938-3564b19025da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.674370 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.674313    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llg4\" (UniqueName: \"kubernetes.io/projected/661c1aad-7c80-40b1-9938-3564b19025da-kube-api-access-7llg4\") pod \"prometheus-k8s-0\" (UID: \"661c1aad-7c80-40b1-9938-3564b19025da\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.790067 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.790045    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:22.923339 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:22.923245    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 07 02:24:22.925937 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:24:22.925900    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661c1aad_7c80_40b1_9938_3564b19025da.slice/crio-e2c700342d5369f3f429c7a18d05fbf4a51703a9899c64eeaee08e3e4eee2859 WatchSource:0}: Error finding container e2c700342d5369f3f429c7a18d05fbf4a51703a9899c64eeaee08e3e4eee2859: Status 404 returned error can't find the container with id e2c700342d5369f3f429c7a18d05fbf4a51703a9899c64eeaee08e3e4eee2859
Jul 07 02:24:23.086757 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:23.086729    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51384e2b-4f02-42f0-a4b0-b99a21730f94" path="/var/lib/kubelet/pods/51384e2b-4f02-42f0-a4b0-b99a21730f94/volumes"
Jul 07 02:24:23.421478 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:23.421448    2570 generic.go:356] "Generic (PLEG): container finished" podID="661c1aad-7c80-40b1-9938-3564b19025da" containerID="3d7f0092a057e16f91c57d9d05b8ba4158dadfcdea4629948a588dd146fab22a" exitCode=0
Jul 07 02:24:23.421626 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:23.421491    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661c1aad-7c80-40b1-9938-3564b19025da","Type":"ContainerDied","Data":"3d7f0092a057e16f91c57d9d05b8ba4158dadfcdea4629948a588dd146fab22a"}
Jul 07 02:24:23.421626 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:23.421540    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661c1aad-7c80-40b1-9938-3564b19025da","Type":"ContainerStarted","Data":"e2c700342d5369f3f429c7a18d05fbf4a51703a9899c64eeaee08e3e4eee2859"}
Jul 07 02:24:24.429208 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:24.429176    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661c1aad-7c80-40b1-9938-3564b19025da","Type":"ContainerStarted","Data":"32f3b2f414bd6b9e963a7b7d723087b52a8844bf8425a3c0a52b44074ce44160"}
Jul 07 02:24:24.429588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:24.429238    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661c1aad-7c80-40b1-9938-3564b19025da","Type":"ContainerStarted","Data":"4e768d750ec157250d3e2ee4a4d7927e76aee9b6af89f56111ac7b94f14f6702"}
Jul 07 02:24:24.429588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:24.429267    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661c1aad-7c80-40b1-9938-3564b19025da","Type":"ContainerStarted","Data":"d8c67dbc0760e7f0f17c9d05d2ab00e95fa395d905f767aea5a987f9aaf0b2f3"}
Jul 07 02:24:24.429588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:24.429297    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661c1aad-7c80-40b1-9938-3564b19025da","Type":"ContainerStarted","Data":"3aa85a0bd2a74856fac5bc06787875944e56d29ccf1c638411be6e7e7caee149"}
Jul 07 02:24:24.429588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:24.429322    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661c1aad-7c80-40b1-9938-3564b19025da","Type":"ContainerStarted","Data":"d913579fedc6fa898db81f1a380928e462e715a8716b8478af230c896d3f6b7e"}
Jul 07 02:24:24.429588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:24.429365    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661c1aad-7c80-40b1-9938-3564b19025da","Type":"ContainerStarted","Data":"7793d9f282f5c20f49dde06514d541388bf54c9d1acfffd19c1ac250d9dd4dbd"}
Jul 07 02:24:24.462327 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:24.462282    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.44483506 podStartE2EDuration="2.46226696s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.4319ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:24:24.459822877 +0000 UTC m=+399.867412578" watchObservedRunningTime="2026-07-07 02:24:24.46226696 +0000 UTC m=+399.869856646"
Jul 07 02:24:27.791038 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:27.791006    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:24:38.266336 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:24:38.266300    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64d5898fc6-9z8vr"]
Jul 07 02:25:03.288321 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.288266    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-64d5898fc6-9z8vr" podUID="b634e258-4ecc-41f0-a799-47781f098c8d" containerName="console" containerID="cri-o://b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c" gracePeriod=15
Jul 07 02:25:03.537254 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.537232    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:25:03.605710 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.605648    2570 generic.go:356] "Generic (PLEG): container finished" podID="b634e258-4ecc-41f0-a799-47781f098c8d" containerID="b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c" exitCode=0
Jul 07 02:25:03.605710 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.605698    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d5898fc6-9z8vr"
Jul 07 02:25:03.605896 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.605723    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d5898fc6-9z8vr" event={"ID":"b634e258-4ecc-41f0-a799-47781f098c8d","Type":"ContainerDied","Data":"b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c"}
Jul 07 02:25:03.605896 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.605762    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d5898fc6-9z8vr" event={"ID":"b634e258-4ecc-41f0-a799-47781f098c8d","Type":"ContainerDied","Data":"71397286f388404e5bbf2fa5963bce621b2d5636878a29f9b974af30d60d8dc9"}
Jul 07 02:25:03.605896 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.605780    2570 scope.go:117] "RemoveContainer" containerID="b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c"
Jul 07 02:25:03.613702 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.613685    2570 scope.go:117] "RemoveContainer" containerID="b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c"
Jul 07 02:25:03.613952 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:25:03.613931    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c\": container with ID starting with b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c not found: ID does not exist" containerID="b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c"
Jul 07 02:25:03.613996 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.613960    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c"} err="failed to get container status \"b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c\": rpc error: code = NotFound desc = could not find container \"b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c\": container with ID starting with b1fe672369f699dc24f8df70e106717e3643c4d491190aa4516b5598aa7baa7c not found: ID does not exist"
Jul 07 02:25:03.693804 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.693774    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-trusted-ca-bundle\") pod \"b634e258-4ecc-41f0-a799-47781f098c8d\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") "
Jul 07 02:25:03.693935 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.693849    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd4l8\" (UniqueName: \"kubernetes.io/projected/b634e258-4ecc-41f0-a799-47781f098c8d-kube-api-access-nd4l8\") pod \"b634e258-4ecc-41f0-a799-47781f098c8d\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") "
Jul 07 02:25:03.694028 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.694010    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-serving-cert\") pod \"b634e258-4ecc-41f0-a799-47781f098c8d\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") "
Jul 07 02:25:03.694092 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.694077    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-console-config\") pod \"b634e258-4ecc-41f0-a799-47781f098c8d\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") "
Jul 07 02:25:03.694129 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.694112    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-oauth-config\") pod \"b634e258-4ecc-41f0-a799-47781f098c8d\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") "
Jul 07 02:25:03.694178 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.694135    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b634e258-4ecc-41f0-a799-47781f098c8d" (UID: "b634e258-4ecc-41f0-a799-47781f098c8d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:25:03.694178 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.694147    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-oauth-serving-cert\") pod \"b634e258-4ecc-41f0-a799-47781f098c8d\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") "
Jul 07 02:25:03.694280 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.694178    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-service-ca\") pod \"b634e258-4ecc-41f0-a799-47781f098c8d\" (UID: \"b634e258-4ecc-41f0-a799-47781f098c8d\") "
Jul 07 02:25:03.694514 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.694490    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-trusted-ca-bundle\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:25:03.694514 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.694498    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-console-config" (OuterVolumeSpecName: "console-config") pod "b634e258-4ecc-41f0-a799-47781f098c8d" (UID: "b634e258-4ecc-41f0-a799-47781f098c8d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:25:03.694679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.694616    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b634e258-4ecc-41f0-a799-47781f098c8d" (UID: "b634e258-4ecc-41f0-a799-47781f098c8d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:25:03.694870 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.694839    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-service-ca" (OuterVolumeSpecName: "service-ca") pod "b634e258-4ecc-41f0-a799-47781f098c8d" (UID: "b634e258-4ecc-41f0-a799-47781f098c8d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:25:03.696244 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.696219    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b634e258-4ecc-41f0-a799-47781f098c8d" (UID: "b634e258-4ecc-41f0-a799-47781f098c8d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:25:03.696321 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.696283    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b634e258-4ecc-41f0-a799-47781f098c8d-kube-api-access-nd4l8" (OuterVolumeSpecName: "kube-api-access-nd4l8") pod "b634e258-4ecc-41f0-a799-47781f098c8d" (UID: "b634e258-4ecc-41f0-a799-47781f098c8d"). InnerVolumeSpecName "kube-api-access-nd4l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:25:03.696385 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.696368    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b634e258-4ecc-41f0-a799-47781f098c8d" (UID: "b634e258-4ecc-41f0-a799-47781f098c8d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:25:03.794849 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.794826    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd4l8\" (UniqueName: \"kubernetes.io/projected/b634e258-4ecc-41f0-a799-47781f098c8d-kube-api-access-nd4l8\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:25:03.794849 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.794848    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-serving-cert\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:25:03.794962 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.794859    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-console-config\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:25:03.794962 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.794867    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b634e258-4ecc-41f0-a799-47781f098c8d-console-oauth-config\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:25:03.794962 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.794875    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-oauth-serving-cert\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:25:03.794962 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.794884    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b634e258-4ecc-41f0-a799-47781f098c8d-service-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:25:03.926381 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.926336    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64d5898fc6-9z8vr"]
Jul 07 02:25:03.932139 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:03.932119    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64d5898fc6-9z8vr"]
Jul 07 02:25:05.085726 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:05.085688    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b634e258-4ecc-41f0-a799-47781f098c8d" path="/var/lib/kubelet/pods/b634e258-4ecc-41f0-a799-47781f098c8d/volumes"
Jul 07 02:25:22.791168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:22.791134    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:25:22.807500 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:22.807476    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:25:23.712362 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:25:23.712312    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 07 02:28:45.004913 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:28:45.004881    2570 scope.go:117] "RemoveContainer" containerID="1cceccc8884b5358046bdeeebe46d0db7e3c4565bd97948393b5eebc778463e4"
Jul 07 02:29:15.137496 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.137464    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lc9nz"]
Jul 07 02:29:15.137865 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.137783    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b634e258-4ecc-41f0-a799-47781f098c8d" containerName="console"
Jul 07 02:29:15.137865 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.137794    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b634e258-4ecc-41f0-a799-47781f098c8d" containerName="console"
Jul 07 02:29:15.137865 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.137848    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b634e258-4ecc-41f0-a799-47781f098c8d" containerName="console"
Jul 07 02:29:15.140493 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.140477    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz"
Jul 07 02:29:15.144461 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.144445    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 07 02:29:15.145080 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.145063    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-h9vzm\""
Jul 07 02:29:15.145122 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.145089    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 07 02:29:15.153167 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.153146    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lc9nz"]
Jul 07 02:29:15.280877 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.280831    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmz8h\" (UniqueName: \"kubernetes.io/projected/27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2-kube-api-access-gmz8h\") pod \"cert-manager-webhook-6486ffd997-lc9nz\" (UID: \"27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz"
Jul 07 02:29:15.281059 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.280977    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lc9nz\" (UID: \"27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz"
Jul 07 02:29:15.382272 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.382238    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lc9nz\" (UID: \"27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz"
Jul 07 02:29:15.382443 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.382275    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmz8h\" (UniqueName: \"kubernetes.io/projected/27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2-kube-api-access-gmz8h\") pod \"cert-manager-webhook-6486ffd997-lc9nz\" (UID: \"27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz"
Jul 07 02:29:15.391230 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.391171    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lc9nz\" (UID: \"27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz"
Jul 07 02:29:15.391326 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.391282    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmz8h\" (UniqueName: \"kubernetes.io/projected/27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2-kube-api-access-gmz8h\") pod \"cert-manager-webhook-6486ffd997-lc9nz\" (UID: \"27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz"
Jul 07 02:29:15.461514 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.461492    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz"
Jul 07 02:29:15.587596 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.587561    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lc9nz"]
Jul 07 02:29:15.590706 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:29:15.590674    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27d86833_bb9d_4a21_8cf1_9fb6e5ae2ac2.slice/crio-3b385a93f58fe941403b55eb93ee802d734d2d898977686a18390ae79991724d WatchSource:0}: Error finding container 3b385a93f58fe941403b55eb93ee802d734d2d898977686a18390ae79991724d: Status 404 returned error can't find the container with id 3b385a93f58fe941403b55eb93ee802d734d2d898977686a18390ae79991724d
Jul 07 02:29:15.592809 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.592787    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:29:15.646380 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:15.646301    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz" event={"ID":"27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2","Type":"ContainerStarted","Data":"3b385a93f58fe941403b55eb93ee802d734d2d898977686a18390ae79991724d"}
Jul 07 02:29:26.675992 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:26.675961    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-chmbw"]
Jul 07 02:29:26.680624 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:26.680602    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-chmbw"
Jul 07 02:29:26.683234 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:26.683215    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wk7ms\""
Jul 07 02:29:26.688445 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:26.688420    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-chmbw"]
Jul 07 02:29:26.768044 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:26.768018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4-bound-sa-token\") pod \"cert-manager-55b4d954d5-chmbw\" (UID: \"ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4\") " pod="cert-manager/cert-manager-55b4d954d5-chmbw"
Jul 07 02:29:26.768154 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:26.768070    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2lzf\" (UniqueName: \"kubernetes.io/projected/ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4-kube-api-access-r2lzf\") pod \"cert-manager-55b4d954d5-chmbw\" (UID: \"ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4\") " pod="cert-manager/cert-manager-55b4d954d5-chmbw"
Jul 07 02:29:26.868760 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:26.868738    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4-bound-sa-token\") pod \"cert-manager-55b4d954d5-chmbw\" (UID: \"ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4\") " pod="cert-manager/cert-manager-55b4d954d5-chmbw"
Jul 07 02:29:26.868861 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:26.868775    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lzf\" (UniqueName: \"kubernetes.io/projected/ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4-kube-api-access-r2lzf\") pod \"cert-manager-55b4d954d5-chmbw\" (UID: \"ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4\") " pod="cert-manager/cert-manager-55b4d954d5-chmbw"
Jul 07 02:29:26.877883 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:26.877852    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4-bound-sa-token\") pod \"cert-manager-55b4d954d5-chmbw\" (UID: \"ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4\") " pod="cert-manager/cert-manager-55b4d954d5-chmbw"
Jul 07 02:29:26.878066 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:26.878044    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lzf\" (UniqueName: \"kubernetes.io/projected/ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4-kube-api-access-r2lzf\") pod \"cert-manager-55b4d954d5-chmbw\" (UID: \"ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4\") " pod="cert-manager/cert-manager-55b4d954d5-chmbw"
Jul 07 02:29:26.990643 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:26.990593    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-chmbw"
Jul 07 02:29:27.120218 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:27.120197    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-chmbw"]
Jul 07 02:29:27.122750 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:29:27.122722    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea39cea0_392f_4a90_a9f7_ab0d5a5ad4b4.slice/crio-7722521bad33a61d2a01350788fa44a164ea877210e8b1831f047b6dccdb641e WatchSource:0}: Error finding container 7722521bad33a61d2a01350788fa44a164ea877210e8b1831f047b6dccdb641e: Status 404 returned error can't find the container with id 7722521bad33a61d2a01350788fa44a164ea877210e8b1831f047b6dccdb641e
Jul 07 02:29:27.697877 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:27.697843    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-chmbw" event={"ID":"ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4","Type":"ContainerStarted","Data":"7722521bad33a61d2a01350788fa44a164ea877210e8b1831f047b6dccdb641e"}
Jul 07 02:29:31.719598 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:31.719523    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-chmbw" event={"ID":"ea39cea0-392f-4a90-a9f7-ab0d5a5ad4b4","Type":"ContainerStarted","Data":"63b6745c9d3aa803eab5618e55831b67649211f33d2fafbc7ea7a2215ac0230e"}
Jul 07 02:29:31.740104 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:31.740059    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-chmbw" podStartSLOduration=1.43102466 podStartE2EDuration="5.74004382s" totalImagesPullingTime="4.30901916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:31.739309122 +0000 UTC m=+707.146898812" watchObservedRunningTime="2026-07-07 02:29:31.74004382 +0000 UTC m=+707.147633506"
Jul 07 02:29:42.769563 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:42.769524    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz" event={"ID":"27d86833-bb9d-4a21-8cf1-9fb6e5ae2ac2","Type":"ContainerStarted","Data":"fc0127003c8f0cec02abc0176646be79b30dfdd4a9fb28376c8fe15b7de8625c"}
Jul 07 02:29:42.770017 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:42.769611    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz"
Jul 07 02:29:42.787632 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:42.787587    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz" podStartSLOduration=1.6506171090000001 podStartE2EDuration="27.787571741s" totalImagesPullingTime="26.136954632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:42.78689404 +0000 UTC m=+718.194483726" watchObservedRunningTime="2026-07-07 02:29:42.787571741 +0000 UTC m=+718.195161428"
Jul 07 02:29:48.775725 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:48.775693    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lc9nz"
Jul 07 02:29:54.927217 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.927180    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"]
Jul 07 02:29:54.933729 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.933709    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:54.937847 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.937805    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lbvft\""
Jul 07 02:29:54.937847 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.937814    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:29:54.938063 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.937872    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 07 02:29:54.938063 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.937853    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 07 02:29:54.938063 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.937848    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 07 02:29:54.938240 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.938225    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 07 02:29:54.947914 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.947892    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"]
Jul 07 02:29:54.989373 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.989331    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/90e7faab-e405-4dd6-a39c-dac8764beaf8-manager-config\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:54.989462 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.989377    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90e7faab-e405-4dd6-a39c-dac8764beaf8-metrics-cert\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:54.989462 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.989408    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90e7faab-e405-4dd6-a39c-dac8764beaf8-cert\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:54.989534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:54.989496    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsxk7\" (UniqueName: \"kubernetes.io/projected/90e7faab-e405-4dd6-a39c-dac8764beaf8-kube-api-access-lsxk7\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:55.090676 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:55.090651    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsxk7\" (UniqueName: \"kubernetes.io/projected/90e7faab-e405-4dd6-a39c-dac8764beaf8-kube-api-access-lsxk7\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:55.090767 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:55.090699    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/90e7faab-e405-4dd6-a39c-dac8764beaf8-manager-config\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:55.090767 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:55.090728    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90e7faab-e405-4dd6-a39c-dac8764beaf8-metrics-cert\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:55.090767 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:55.090752    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90e7faab-e405-4dd6-a39c-dac8764beaf8-cert\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:55.091338 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:55.091319    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/90e7faab-e405-4dd6-a39c-dac8764beaf8-manager-config\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:55.093396 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:55.093379    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90e7faab-e405-4dd6-a39c-dac8764beaf8-metrics-cert\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:55.093474 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:55.093439    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90e7faab-e405-4dd6-a39c-dac8764beaf8-cert\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:55.112319 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:55.112301    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsxk7\" (UniqueName: \"kubernetes.io/projected/90e7faab-e405-4dd6-a39c-dac8764beaf8-kube-api-access-lsxk7\") pod \"lws-controller-manager-d5cdd5cf7-9fjw7\" (UID: \"90e7faab-e405-4dd6-a39c-dac8764beaf8\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:55.242599 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:55.242547    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:55.376819 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:55.376793    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"]
Jul 07 02:29:55.379060 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:29:55.379037    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e7faab_e405_4dd6_a39c_dac8764beaf8.slice/crio-90bedd0f339322dea5574c75e1c9c0f7c68b482a41436f149094ee56134cd068 WatchSource:0}: Error finding container 90bedd0f339322dea5574c75e1c9c0f7c68b482a41436f149094ee56134cd068: Status 404 returned error can't find the container with id 90bedd0f339322dea5574c75e1c9c0f7c68b482a41436f149094ee56134cd068
Jul 07 02:29:55.828150 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:55.828119    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7" event={"ID":"90e7faab-e405-4dd6-a39c-dac8764beaf8","Type":"ContainerStarted","Data":"90bedd0f339322dea5574c75e1c9c0f7c68b482a41436f149094ee56134cd068"}
Jul 07 02:29:59.847437 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:59.847401    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7" event={"ID":"90e7faab-e405-4dd6-a39c-dac8764beaf8","Type":"ContainerStarted","Data":"c9651bf4a922b11f8e3f56f207817dcf11bee171d88d1ca67d46a14f0c65f1d3"}
Jul 07 02:29:59.847842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:59.847469    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:29:59.865246 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:29:59.865201    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7" podStartSLOduration=2.259907993 podStartE2EDuration="5.865186577s" totalImagesPullingTime="3.605278584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:59.863469094 +0000 UTC m=+735.271058989" watchObservedRunningTime="2026-07-07 02:29:59.865186577 +0000 UTC m=+735.272776263"
Jul 07 02:30:10.853453 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:10.853419    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-9fjw7"
Jul 07 02:30:29.916949 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:29.916921    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj"]
Jul 07 02:30:29.921842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:29.921824    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj"
Jul 07 02:30:29.924555 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:29.924535    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wswbs\""
Jul 07 02:30:29.924555 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:29.924555    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 07 02:30:29.924744 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:29.924608    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 02:30:29.924795 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:29.924742    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 02:30:29.931220 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:29.931198    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj"]
Jul 07 02:30:29.969153 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:29.969122    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wps4\" (UniqueName: \"kubernetes.io/projected/f0f0bd37-9814-48fb-8f9e-76480fc16649-kube-api-access-4wps4\") pod \"dns-operator-controller-manager-65b49595d7-2qhqj\" (UID: \"f0f0bd37-9814-48fb-8f9e-76480fc16649\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj"
Jul 07 02:30:30.070230 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:30.070206    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wps4\" (UniqueName: \"kubernetes.io/projected/f0f0bd37-9814-48fb-8f9e-76480fc16649-kube-api-access-4wps4\") pod \"dns-operator-controller-manager-65b49595d7-2qhqj\" (UID: \"f0f0bd37-9814-48fb-8f9e-76480fc16649\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj"
Jul 07 02:30:30.079668 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:30.079648    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wps4\" (UniqueName: \"kubernetes.io/projected/f0f0bd37-9814-48fb-8f9e-76480fc16649-kube-api-access-4wps4\") pod \"dns-operator-controller-manager-65b49595d7-2qhqj\" (UID: \"f0f0bd37-9814-48fb-8f9e-76480fc16649\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj"
Jul 07 02:30:30.232434 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:30.232373    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj"
Jul 07 02:30:30.376278 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:30.376245    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj"]
Jul 07 02:30:30.379921 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:30:30.379892    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f0bd37_9814_48fb_8f9e_76480fc16649.slice/crio-cf847278c759734bd0bc552588d0bde8990699050214512c3a0e77d23b727d1e WatchSource:0}: Error finding container cf847278c759734bd0bc552588d0bde8990699050214512c3a0e77d23b727d1e: Status 404 returned error can't find the container with id cf847278c759734bd0bc552588d0bde8990699050214512c3a0e77d23b727d1e
Jul 07 02:30:30.974765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:30.974728    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj" event={"ID":"f0f0bd37-9814-48fb-8f9e-76480fc16649","Type":"ContainerStarted","Data":"cf847278c759734bd0bc552588d0bde8990699050214512c3a0e77d23b727d1e"}
Jul 07 02:30:32.734799 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:32.734757    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-shqc4"]
Jul 07 02:30:32.742703 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:32.742676    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-shqc4"
Jul 07 02:30:32.745804 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:32.745778    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-w6v6w\""
Jul 07 02:30:32.753944 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:32.753888    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-shqc4"]
Jul 07 02:30:32.790738 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:32.790712    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcq5d\" (UniqueName: \"kubernetes.io/projected/d37031cb-6a87-4a89-96e7-3280ac6be8ee-kube-api-access-bcq5d\") pod \"authorino-operator-6d85f6564-shqc4\" (UID: \"d37031cb-6a87-4a89-96e7-3280ac6be8ee\") " pod="kuadrant-system/authorino-operator-6d85f6564-shqc4"
Jul 07 02:30:32.891943 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:32.891912    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcq5d\" (UniqueName: \"kubernetes.io/projected/d37031cb-6a87-4a89-96e7-3280ac6be8ee-kube-api-access-bcq5d\") pod \"authorino-operator-6d85f6564-shqc4\" (UID: \"d37031cb-6a87-4a89-96e7-3280ac6be8ee\") " pod="kuadrant-system/authorino-operator-6d85f6564-shqc4"
Jul 07 02:30:32.901550 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:32.901510    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcq5d\" (UniqueName: \"kubernetes.io/projected/d37031cb-6a87-4a89-96e7-3280ac6be8ee-kube-api-access-bcq5d\") pod \"authorino-operator-6d85f6564-shqc4\" (UID: \"d37031cb-6a87-4a89-96e7-3280ac6be8ee\") " pod="kuadrant-system/authorino-operator-6d85f6564-shqc4"
Jul 07 02:30:33.056721 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:33.056677    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-shqc4"
Jul 07 02:30:33.488118 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:33.488091    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-shqc4"]
Jul 07 02:30:33.490466 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:30:33.490440    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37031cb_6a87_4a89_96e7_3280ac6be8ee.slice/crio-d3850b777c8814d840abd96a5761b88191220089089c3e31e77556d0839735e0 WatchSource:0}: Error finding container d3850b777c8814d840abd96a5761b88191220089089c3e31e77556d0839735e0: Status 404 returned error can't find the container with id d3850b777c8814d840abd96a5761b88191220089089c3e31e77556d0839735e0
Jul 07 02:30:33.994618 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:33.994587    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj" event={"ID":"f0f0bd37-9814-48fb-8f9e-76480fc16649","Type":"ContainerStarted","Data":"d2fc35854160066597ab7b37856154bcc198fe7f17d1e810fcee2a0bce662d41"}
Jul 07 02:30:33.995051 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:33.994701    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj"
Jul 07 02:30:33.996041 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:33.996021    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-shqc4" event={"ID":"d37031cb-6a87-4a89-96e7-3280ac6be8ee","Type":"ContainerStarted","Data":"d3850b777c8814d840abd96a5761b88191220089089c3e31e77556d0839735e0"}
Jul 07 02:30:34.020245 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:34.020198    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj" podStartSLOduration=1.983307515 podStartE2EDuration="5.020185636s" totalImagesPullingTime="3.036878121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:30:34.018638516 +0000 UTC m=+769.426228199" watchObservedRunningTime="2026-07-07 02:30:34.020185636 +0000 UTC m=+769.427775322"
Jul 07 02:30:37.013487 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:37.013460    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-shqc4" event={"ID":"d37031cb-6a87-4a89-96e7-3280ac6be8ee","Type":"ContainerStarted","Data":"a676a7222bbb326dd46005d718c635da38ee00cfd7fae1b0a462c823c4ce5187"}
Jul 07 02:30:37.013825 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:37.013575    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6d85f6564-shqc4"
Jul 07 02:30:37.044713 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:37.044672    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6d85f6564-shqc4" podStartSLOduration=1.732276514 podStartE2EDuration="5.044658473s" totalImagesPullingTime="3.312381959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:30:37.043405615 +0000 UTC m=+772.450995301" watchObservedRunningTime="2026-07-07 02:30:37.044658473 +0000 UTC m=+772.452248160"
Jul 07 02:30:45.003760 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:45.003729    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-2qhqj"
Jul 07 02:30:48.019704 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:30:48.019676    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6d85f6564-shqc4"
Jul 07 02:31:20.550306 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.550274    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"]
Jul 07 02:31:20.552553 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.552537    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:20.555069 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.555036    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 07 02:31:20.555172 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.555125    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nvhdn\""
Jul 07 02:31:20.563008 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.562983    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"]
Jul 07 02:31:20.650301 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.650270    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"]
Jul 07 02:31:20.669842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.669816    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx6bt\" (UniqueName: \"kubernetes.io/projected/3452222b-28e9-47e9-ba51-c0e16f9dc437-kube-api-access-hx6bt\") pod \"limitador-limitador-ff68ff5cb-tffrc\" (UID: \"3452222b-28e9-47e9-ba51-c0e16f9dc437\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:20.669967 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.669878    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3452222b-28e9-47e9-ba51-c0e16f9dc437-config-file\") pod \"limitador-limitador-ff68ff5cb-tffrc\" (UID: \"3452222b-28e9-47e9-ba51-c0e16f9dc437\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:20.770730 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.770707    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6bt\" (UniqueName: \"kubernetes.io/projected/3452222b-28e9-47e9-ba51-c0e16f9dc437-kube-api-access-hx6bt\") pod \"limitador-limitador-ff68ff5cb-tffrc\" (UID: \"3452222b-28e9-47e9-ba51-c0e16f9dc437\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:20.770840 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.770767    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3452222b-28e9-47e9-ba51-c0e16f9dc437-config-file\") pod \"limitador-limitador-ff68ff5cb-tffrc\" (UID: \"3452222b-28e9-47e9-ba51-c0e16f9dc437\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:20.771313 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.771297    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3452222b-28e9-47e9-ba51-c0e16f9dc437-config-file\") pod \"limitador-limitador-ff68ff5cb-tffrc\" (UID: \"3452222b-28e9-47e9-ba51-c0e16f9dc437\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:20.779450 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.779422    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6bt\" (UniqueName: \"kubernetes.io/projected/3452222b-28e9-47e9-ba51-c0e16f9dc437-kube-api-access-hx6bt\") pod \"limitador-limitador-ff68ff5cb-tffrc\" (UID: \"3452222b-28e9-47e9-ba51-c0e16f9dc437\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:20.864515 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:20.864460    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:21.015812 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:21.015788    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"]
Jul 07 02:31:21.018193 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:31:21.018161    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3452222b_28e9_47e9_ba51_c0e16f9dc437.slice/crio-8b253ffc2405796b4f23cd1c04b8766a14d52230d81a9d783b940242f0247632 WatchSource:0}: Error finding container 8b253ffc2405796b4f23cd1c04b8766a14d52230d81a9d783b940242f0247632: Status 404 returned error can't find the container with id 8b253ffc2405796b4f23cd1c04b8766a14d52230d81a9d783b940242f0247632
Jul 07 02:31:21.225000 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:21.224920    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc" event={"ID":"3452222b-28e9-47e9-ba51-c0e16f9dc437","Type":"ContainerStarted","Data":"8b253ffc2405796b4f23cd1c04b8766a14d52230d81a9d783b940242f0247632"}
Jul 07 02:31:26.256604 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:26.256571    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc" event={"ID":"3452222b-28e9-47e9-ba51-c0e16f9dc437","Type":"ContainerStarted","Data":"658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a"}
Jul 07 02:31:26.256975 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:26.256708    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:26.277293 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:26.277247    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc" podStartSLOduration=1.57794818 podStartE2EDuration="6.277232227s" totalImagesPullingTime="4.699284047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:31:26.275774533 +0000 UTC m=+821.683364233" watchObservedRunningTime="2026-07-07 02:31:26.277232227 +0000 UTC m=+821.684821914"
Jul 07 02:31:37.130728 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:37.130696    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"]
Jul 07 02:31:37.131165 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:37.130876    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc" podUID="3452222b-28e9-47e9-ba51-c0e16f9dc437" containerName="limitador" containerID="cri-o://658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a" gracePeriod=30
Jul 07 02:31:37.131610 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:37.131591    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:37.673305 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:37.673281    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:37.703263 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:37.703214    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx6bt\" (UniqueName: \"kubernetes.io/projected/3452222b-28e9-47e9-ba51-c0e16f9dc437-kube-api-access-hx6bt\") pod \"3452222b-28e9-47e9-ba51-c0e16f9dc437\" (UID: \"3452222b-28e9-47e9-ba51-c0e16f9dc437\") "
Jul 07 02:31:37.703376 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:37.703300    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3452222b-28e9-47e9-ba51-c0e16f9dc437-config-file\") pod \"3452222b-28e9-47e9-ba51-c0e16f9dc437\" (UID: \"3452222b-28e9-47e9-ba51-c0e16f9dc437\") "
Jul 07 02:31:37.703699 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:37.703672    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3452222b-28e9-47e9-ba51-c0e16f9dc437-config-file" (OuterVolumeSpecName: "config-file") pod "3452222b-28e9-47e9-ba51-c0e16f9dc437" (UID: "3452222b-28e9-47e9-ba51-c0e16f9dc437"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:31:37.705415 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:37.705395    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3452222b-28e9-47e9-ba51-c0e16f9dc437-kube-api-access-hx6bt" (OuterVolumeSpecName: "kube-api-access-hx6bt") pod "3452222b-28e9-47e9-ba51-c0e16f9dc437" (UID: "3452222b-28e9-47e9-ba51-c0e16f9dc437"). InnerVolumeSpecName "kube-api-access-hx6bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:31:37.804048 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:37.804026    2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3452222b-28e9-47e9-ba51-c0e16f9dc437-config-file\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:31:37.804048 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:37.804048    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx6bt\" (UniqueName: \"kubernetes.io/projected/3452222b-28e9-47e9-ba51-c0e16f9dc437-kube-api-access-hx6bt\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:31:38.311319 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:38.311283    2570 generic.go:356] "Generic (PLEG): container finished" podID="3452222b-28e9-47e9-ba51-c0e16f9dc437" containerID="658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a" exitCode=0
Jul 07 02:31:38.311715 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:38.311327    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"
Jul 07 02:31:38.311715 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:38.311336    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc" event={"ID":"3452222b-28e9-47e9-ba51-c0e16f9dc437","Type":"ContainerDied","Data":"658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a"}
Jul 07 02:31:38.311715 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:38.311406    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-tffrc" event={"ID":"3452222b-28e9-47e9-ba51-c0e16f9dc437","Type":"ContainerDied","Data":"8b253ffc2405796b4f23cd1c04b8766a14d52230d81a9d783b940242f0247632"}
Jul 07 02:31:38.311715 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:38.311430    2570 scope.go:117] "RemoveContainer" containerID="658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a"
Jul 07 02:31:38.322983 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:38.322960    2570 scope.go:117] "RemoveContainer" containerID="658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a"
Jul 07 02:31:38.323428 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:31:38.323402    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a\": container with ID starting with 658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a not found: ID does not exist" containerID="658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a"
Jul 07 02:31:38.323539 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:38.323438    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a"} err="failed to get container status \"658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a\": rpc error: code = NotFound desc = could not find container \"658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a\": container with ID starting with 658c28dedfdebcd7a260d948f3b4e1d2a74823e89dd1ac14b8e853d2da0c173a not found: ID does not exist"
Jul 07 02:31:38.332286 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:38.332263    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"]
Jul 07 02:31:38.336292 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:38.336271    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-tffrc"]
Jul 07 02:31:39.086734 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:31:39.086700    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3452222b-28e9-47e9-ba51-c0e16f9dc437" path="/var/lib/kubelet/pods/3452222b-28e9-47e9-ba51-c0e16f9dc437/volumes"
Jul 07 02:32:26.050927 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.050886    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-ngwbc"]
Jul 07 02:32:26.051843 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.051816    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3452222b-28e9-47e9-ba51-c0e16f9dc437" containerName="limitador"
Jul 07 02:32:26.051843 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.051843    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3452222b-28e9-47e9-ba51-c0e16f9dc437" containerName="limitador"
Jul 07 02:32:26.052022 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.051999    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3452222b-28e9-47e9-ba51-c0e16f9dc437" containerName="limitador"
Jul 07 02:32:26.055958 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.055935    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-ngwbc"
Jul 07 02:32:26.059200 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.059183    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 07 02:32:26.059450 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.059434    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-8x2c8\""
Jul 07 02:32:26.059517 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.059489    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 02:32:26.060119 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.060100    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 02:32:26.064392 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.064229    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-ngwbc"]
Jul 07 02:32:26.069586 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.069566    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-m82hl"]
Jul 07 02:32:26.072834 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.072816    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-m82hl"
Jul 07 02:32:26.075694 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.075675    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 07 02:32:26.075974 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.075753    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-kdvrh\""
Jul 07 02:32:26.086583 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.086564    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-m82hl"]
Jul 07 02:32:26.183870 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.183839    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ca53f9-9f2d-4c40-a340-dcf857dc3f00-tls-certs\") pod \"model-serving-api-fd65d7d6-ngwbc\" (UID: \"42ca53f9-9f2d-4c40-a340-dcf857dc3f00\") " pod="kserve/model-serving-api-fd65d7d6-ngwbc"
Jul 07 02:32:26.184035 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.183881    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djxbx\" (UniqueName: \"kubernetes.io/projected/ea806b72-e957-42cd-9348-35c377e6c42a-kube-api-access-djxbx\") pod \"odh-model-controller-84b54b7f97-m82hl\" (UID: \"ea806b72-e957-42cd-9348-35c377e6c42a\") " pod="kserve/odh-model-controller-84b54b7f97-m82hl"
Jul 07 02:32:26.184035 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.183904    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea806b72-e957-42cd-9348-35c377e6c42a-cert\") pod \"odh-model-controller-84b54b7f97-m82hl\" (UID: \"ea806b72-e957-42cd-9348-35c377e6c42a\") " pod="kserve/odh-model-controller-84b54b7f97-m82hl"
Jul 07 02:32:26.184035 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.183989    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6bm\" (UniqueName: \"kubernetes.io/projected/42ca53f9-9f2d-4c40-a340-dcf857dc3f00-kube-api-access-wn6bm\") pod \"model-serving-api-fd65d7d6-ngwbc\" (UID: \"42ca53f9-9f2d-4c40-a340-dcf857dc3f00\") " pod="kserve/model-serving-api-fd65d7d6-ngwbc"
Jul 07 02:32:26.285018 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.284996    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ca53f9-9f2d-4c40-a340-dcf857dc3f00-tls-certs\") pod \"model-serving-api-fd65d7d6-ngwbc\" (UID: \"42ca53f9-9f2d-4c40-a340-dcf857dc3f00\") " pod="kserve/model-serving-api-fd65d7d6-ngwbc"
Jul 07 02:32:26.285141 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.285033    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djxbx\" (UniqueName: \"kubernetes.io/projected/ea806b72-e957-42cd-9348-35c377e6c42a-kube-api-access-djxbx\") pod \"odh-model-controller-84b54b7f97-m82hl\" (UID: \"ea806b72-e957-42cd-9348-35c377e6c42a\") " pod="kserve/odh-model-controller-84b54b7f97-m82hl"
Jul 07 02:32:26.285141 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.285050    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea806b72-e957-42cd-9348-35c377e6c42a-cert\") pod \"odh-model-controller-84b54b7f97-m82hl\" (UID: \"ea806b72-e957-42cd-9348-35c377e6c42a\") " pod="kserve/odh-model-controller-84b54b7f97-m82hl"
Jul 07 02:32:26.285141 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.285101    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6bm\" (UniqueName: \"kubernetes.io/projected/42ca53f9-9f2d-4c40-a340-dcf857dc3f00-kube-api-access-wn6bm\") pod \"model-serving-api-fd65d7d6-ngwbc\" (UID: \"42ca53f9-9f2d-4c40-a340-dcf857dc3f00\") " pod="kserve/model-serving-api-fd65d7d6-ngwbc"
Jul 07 02:32:26.285268 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:32:26.285168    2570 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 07 02:32:26.285268 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:32:26.285262    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ca53f9-9f2d-4c40-a340-dcf857dc3f00-tls-certs podName:42ca53f9-9f2d-4c40-a340-dcf857dc3f00 nodeName:}" failed. No retries permitted until 2026-07-07 02:32:26.785241902 +0000 UTC m=+882.192831570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/42ca53f9-9f2d-4c40-a340-dcf857dc3f00-tls-certs") pod "model-serving-api-fd65d7d6-ngwbc" (UID: "42ca53f9-9f2d-4c40-a340-dcf857dc3f00") : secret "model-serving-api-tls" not found
Jul 07 02:32:26.287687 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.287669    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea806b72-e957-42cd-9348-35c377e6c42a-cert\") pod \"odh-model-controller-84b54b7f97-m82hl\" (UID: \"ea806b72-e957-42cd-9348-35c377e6c42a\") " pod="kserve/odh-model-controller-84b54b7f97-m82hl"
Jul 07 02:32:26.293629 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.293611    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6bm\" (UniqueName: \"kubernetes.io/projected/42ca53f9-9f2d-4c40-a340-dcf857dc3f00-kube-api-access-wn6bm\") pod \"model-serving-api-fd65d7d6-ngwbc\" (UID: \"42ca53f9-9f2d-4c40-a340-dcf857dc3f00\") " pod="kserve/model-serving-api-fd65d7d6-ngwbc"
Jul 07 02:32:26.294015 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.293994    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djxbx\" (UniqueName: \"kubernetes.io/projected/ea806b72-e957-42cd-9348-35c377e6c42a-kube-api-access-djxbx\") pod \"odh-model-controller-84b54b7f97-m82hl\" (UID: \"ea806b72-e957-42cd-9348-35c377e6c42a\") " pod="kserve/odh-model-controller-84b54b7f97-m82hl"
Jul 07 02:32:26.382092 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.382041    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-m82hl"
Jul 07 02:32:26.514124 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.514097    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-m82hl"]
Jul 07 02:32:26.517521 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:32:26.517485    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea806b72_e957_42cd_9348_35c377e6c42a.slice/crio-9c91b4d3c5cd9517f4dbadbfad7d6e16144fd422c4a881b09cb1ac506c8afe7d WatchSource:0}: Error finding container 9c91b4d3c5cd9517f4dbadbfad7d6e16144fd422c4a881b09cb1ac506c8afe7d: Status 404 returned error can't find the container with id 9c91b4d3c5cd9517f4dbadbfad7d6e16144fd422c4a881b09cb1ac506c8afe7d
Jul 07 02:32:26.536418 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.536392    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-m82hl" event={"ID":"ea806b72-e957-42cd-9348-35c377e6c42a","Type":"ContainerStarted","Data":"9c91b4d3c5cd9517f4dbadbfad7d6e16144fd422c4a881b09cb1ac506c8afe7d"}
Jul 07 02:32:26.790300 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.790275    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ca53f9-9f2d-4c40-a340-dcf857dc3f00-tls-certs\") pod \"model-serving-api-fd65d7d6-ngwbc\" (UID: \"42ca53f9-9f2d-4c40-a340-dcf857dc3f00\") " pod="kserve/model-serving-api-fd65d7d6-ngwbc"
Jul 07 02:32:26.792718 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.792700    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42ca53f9-9f2d-4c40-a340-dcf857dc3f00-tls-certs\") pod \"model-serving-api-fd65d7d6-ngwbc\" (UID: \"42ca53f9-9f2d-4c40-a340-dcf857dc3f00\") " pod="kserve/model-serving-api-fd65d7d6-ngwbc"
Jul 07 02:32:26.968171 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:26.968150    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-ngwbc"
Jul 07 02:32:27.100199 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:27.100177    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-ngwbc"]
Jul 07 02:32:27.102754 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:32:27.102729    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ca53f9_9f2d_4c40_a340_dcf857dc3f00.slice/crio-ca6af618ef684a83ee1d36d683bccfd90fd874283a756233131ed023c5b3f1f1 WatchSource:0}: Error finding container ca6af618ef684a83ee1d36d683bccfd90fd874283a756233131ed023c5b3f1f1: Status 404 returned error can't find the container with id ca6af618ef684a83ee1d36d683bccfd90fd874283a756233131ed023c5b3f1f1
Jul 07 02:32:27.544570 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:27.544533    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-ngwbc" event={"ID":"42ca53f9-9f2d-4c40-a340-dcf857dc3f00","Type":"ContainerStarted","Data":"ca6af618ef684a83ee1d36d683bccfd90fd874283a756233131ed023c5b3f1f1"}
Jul 07 02:32:30.564581 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:30.564543    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-m82hl" event={"ID":"ea806b72-e957-42cd-9348-35c377e6c42a","Type":"ContainerStarted","Data":"5858c99c7091b7a3926185eca0b1dfb018140664db4a6bf1566c66b69de49fe2"}
Jul 07 02:32:30.565371 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:30.565321    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-m82hl"
Jul 07 02:32:30.567178 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:30.567147    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-ngwbc" event={"ID":"42ca53f9-9f2d-4c40-a340-dcf857dc3f00","Type":"ContainerStarted","Data":"1aa78dc0f92274556861bd5d2f14223ec7060991b8fe9863b204174497934a85"}
Jul 07 02:32:30.567579 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:30.567555    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-ngwbc"
Jul 07 02:32:30.582287 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:30.582245    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-m82hl" podStartSLOduration=1.083403696 podStartE2EDuration="4.582231121s" totalImagesPullingTime="3.498827425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:32:30.5819273 +0000 UTC m=+885.989516993" watchObservedRunningTime="2026-07-07 02:32:30.582231121 +0000 UTC m=+885.989820808"
Jul 07 02:32:30.600537 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:30.600494    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-ngwbc" podStartSLOduration=1.690074128 podStartE2EDuration="4.600480532s" totalImagesPullingTime="2.910406404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:32:30.599512068 +0000 UTC m=+886.007101778" watchObservedRunningTime="2026-07-07 02:32:30.600480532 +0000 UTC m=+886.008070219"
Jul 07 02:32:41.576477 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:41.576405    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-ngwbc"
Jul 07 02:32:42.579157 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:42.579124    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-m82hl"
Jul 07 02:32:43.376543 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.376510    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-g9h6p"]
Jul 07 02:32:43.380068 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.380053    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g9h6p"
Jul 07 02:32:43.382541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.382520    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 02:32:43.382672 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.382555    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-zl4w7\""
Jul 07 02:32:43.388453 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.388419    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-g9h6p"]
Jul 07 02:32:43.430828 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.430803    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e242789-fd76-4af8-967f-6de77e92d596-model-cache\") pod \"s3-init-g9h6p\" (UID: \"9e242789-fd76-4af8-967f-6de77e92d596\") " pod="kserve/s3-init-g9h6p"
Jul 07 02:32:43.430929 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.430839    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45h7\" (UniqueName: \"kubernetes.io/projected/9e242789-fd76-4af8-967f-6de77e92d596-kube-api-access-g45h7\") pod \"s3-init-g9h6p\" (UID: \"9e242789-fd76-4af8-967f-6de77e92d596\") " pod="kserve/s3-init-g9h6p"
Jul 07 02:32:43.531379 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.531335    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e242789-fd76-4af8-967f-6de77e92d596-model-cache\") pod \"s3-init-g9h6p\" (UID: \"9e242789-fd76-4af8-967f-6de77e92d596\") " pod="kserve/s3-init-g9h6p"
Jul 07 02:32:43.531461 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.531401    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45h7\" (UniqueName: \"kubernetes.io/projected/9e242789-fd76-4af8-967f-6de77e92d596-kube-api-access-g45h7\") pod \"s3-init-g9h6p\" (UID: \"9e242789-fd76-4af8-967f-6de77e92d596\") " pod="kserve/s3-init-g9h6p"
Jul 07 02:32:43.531693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.531678    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e242789-fd76-4af8-967f-6de77e92d596-model-cache\") pod \"s3-init-g9h6p\" (UID: \"9e242789-fd76-4af8-967f-6de77e92d596\") " pod="kserve/s3-init-g9h6p"
Jul 07 02:32:43.544527 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.544502    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45h7\" (UniqueName: \"kubernetes.io/projected/9e242789-fd76-4af8-967f-6de77e92d596-kube-api-access-g45h7\") pod \"s3-init-g9h6p\" (UID: \"9e242789-fd76-4af8-967f-6de77e92d596\") " pod="kserve/s3-init-g9h6p"
Jul 07 02:32:43.690632 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.690573    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g9h6p"
Jul 07 02:32:43.820090 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:43.820065    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-g9h6p"]
Jul 07 02:32:43.822711 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:32:43.822689    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e242789_fd76_4af8_967f_6de77e92d596.slice/crio-82381ca13229f1b6e35bc34f3720ed81c31b612e52f1e3a4726b48697af90b52 WatchSource:0}: Error finding container 82381ca13229f1b6e35bc34f3720ed81c31b612e52f1e3a4726b48697af90b52: Status 404 returned error can't find the container with id 82381ca13229f1b6e35bc34f3720ed81c31b612e52f1e3a4726b48697af90b52
Jul 07 02:32:44.636790 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:44.636753    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g9h6p" event={"ID":"9e242789-fd76-4af8-967f-6de77e92d596","Type":"ContainerStarted","Data":"82381ca13229f1b6e35bc34f3720ed81c31b612e52f1e3a4726b48697af90b52"}
Jul 07 02:32:47.652989 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:47.652950    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g9h6p" event={"ID":"9e242789-fd76-4af8-967f-6de77e92d596","Type":"ContainerStarted","Data":"4bbd2868aaecf4563bd43721d62801847801f9aa4cae76f07643832b16a533a5"}
Jul 07 02:32:49.663805 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:49.663767    2570 generic.go:356] "Generic (PLEG): container finished" podID="9e242789-fd76-4af8-967f-6de77e92d596" containerID="4bbd2868aaecf4563bd43721d62801847801f9aa4cae76f07643832b16a533a5" exitCode=0
Jul 07 02:32:49.663805 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:49.663807    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g9h6p" event={"ID":"9e242789-fd76-4af8-967f-6de77e92d596","Type":"ContainerDied","Data":"4bbd2868aaecf4563bd43721d62801847801f9aa4cae76f07643832b16a533a5"}
Jul 07 02:32:54.343068 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:54.343044    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 02:32:54.694230 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:54.694154    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g9h6p" event={"ID":"9e242789-fd76-4af8-967f-6de77e92d596","Type":"ContainerStarted","Data":"551817b6b9c42e486c90c64c1fb62ce54e2e7be32490431192ade5ea2dea5898"}
Jul 07 02:32:54.715421 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:32:54.715375    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-g9h6p" podStartSLOduration=1.82786118 podStartE2EDuration="11.715362411s" totalImagesPullingTime="8.339384943s" totalInitContainerRuntime="1.548116288s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:32:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:32:54.713458826 +0000 UTC m=+910.121048514" watchObservedRunningTime="2026-07-07 02:32:54.715362411 +0000 UTC m=+910.122952091"
Jul 07 02:34:00.007371 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:00.007312    2570 generic.go:356] "Generic (PLEG): container finished" podID="9e242789-fd76-4af8-967f-6de77e92d596" containerID="551817b6b9c42e486c90c64c1fb62ce54e2e7be32490431192ade5ea2dea5898" exitCode=0
Jul 07 02:34:00.008035 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:00.007385    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g9h6p" event={"ID":"9e242789-fd76-4af8-967f-6de77e92d596","Type":"ContainerDied","Data":"551817b6b9c42e486c90c64c1fb62ce54e2e7be32490431192ade5ea2dea5898"}
Jul 07 02:34:01.157083 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:01.157062    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g9h6p"
Jul 07 02:34:01.244702 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:01.244675    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e242789-fd76-4af8-967f-6de77e92d596-model-cache\") pod \"9e242789-fd76-4af8-967f-6de77e92d596\" (UID: \"9e242789-fd76-4af8-967f-6de77e92d596\") "
Jul 07 02:34:01.244838 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:01.244750    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g45h7\" (UniqueName: \"kubernetes.io/projected/9e242789-fd76-4af8-967f-6de77e92d596-kube-api-access-g45h7\") pod \"9e242789-fd76-4af8-967f-6de77e92d596\" (UID: \"9e242789-fd76-4af8-967f-6de77e92d596\") "
Jul 07 02:34:01.247041 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:01.247011    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e242789-fd76-4af8-967f-6de77e92d596-kube-api-access-g45h7" (OuterVolumeSpecName: "kube-api-access-g45h7") pod "9e242789-fd76-4af8-967f-6de77e92d596" (UID: "9e242789-fd76-4af8-967f-6de77e92d596"). InnerVolumeSpecName "kube-api-access-g45h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:34:01.263248 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:01.263223    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e242789-fd76-4af8-967f-6de77e92d596-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e242789-fd76-4af8-967f-6de77e92d596" (UID: "9e242789-fd76-4af8-967f-6de77e92d596"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:34:01.345583 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:01.345560    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45h7\" (UniqueName: \"kubernetes.io/projected/9e242789-fd76-4af8-967f-6de77e92d596-kube-api-access-g45h7\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:34:01.345583 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:01.345583    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e242789-fd76-4af8-967f-6de77e92d596-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:34:02.020410 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:02.020380    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-g9h6p" event={"ID":"9e242789-fd76-4af8-967f-6de77e92d596","Type":"ContainerDied","Data":"82381ca13229f1b6e35bc34f3720ed81c31b612e52f1e3a4726b48697af90b52"}
Jul 07 02:34:02.020410 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:02.020399    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-g9h6p"
Jul 07 02:34:02.020604 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:02.020423    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82381ca13229f1b6e35bc34f3720ed81c31b612e52f1e3a4726b48697af90b52"
Jul 07 02:34:14.979529 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.979451    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"]
Jul 07 02:34:14.979868 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.979807    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e242789-fd76-4af8-967f-6de77e92d596" containerName="download-hf-model"
Jul 07 02:34:14.979868 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.979818    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e242789-fd76-4af8-967f-6de77e92d596" containerName="download-hf-model"
Jul 07 02:34:14.979868 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.979833    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e242789-fd76-4af8-967f-6de77e92d596" containerName="s3-init"
Jul 07 02:34:14.979868 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.979839    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e242789-fd76-4af8-967f-6de77e92d596" containerName="s3-init"
Jul 07 02:34:14.979995 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.979897    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e242789-fd76-4af8-967f-6de77e92d596" containerName="s3-init"
Jul 07 02:34:14.983004 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.982985    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:14.985386 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.985339    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-zq6fc\""
Jul 07 02:34:14.985497 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.985415    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:34:14.985569 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.985525    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 07 02:34:14.985790 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.985778    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:34:14.995094 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:14.995073    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"]
Jul 07 02:34:15.049226 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.049200    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.049326 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.049245    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4srj\" (UniqueName: \"kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-kube-api-access-r4srj\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.049399 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.049330    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.049437 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.049396    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24dd9a67-4b60-4e98-917d-e178435cfc18-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.049437 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.049421    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.049500 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.049437    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.049500 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.049479    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.049603 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.049508    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.049603 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.049539    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.149992 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.149968    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150090 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150003    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24dd9a67-4b60-4e98-917d-e178435cfc18-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150090 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150023    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150179 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150156    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150252 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150224    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150389 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150268    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150389 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150314    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150389 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150358    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150389 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150341    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150606 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150457    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4srj\" (UniqueName: \"kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-kube-api-access-r4srj\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150742 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150719    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150873 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150857    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150934 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150890    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.150934 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.150915    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24dd9a67-4b60-4e98-917d-e178435cfc18-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.152943 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.152922    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.153050 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.153033    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.162664 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.162636    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.163161 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.163145    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4srj\" (UniqueName: \"kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-kube-api-access-r4srj\") pod \"router-gateway-1-openshift-default-54c47cdfc7-dlzm4\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.293166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.293147    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:15.436484 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:15.436454    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"]
Jul 07 02:34:15.438249 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:34:15.438221    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24dd9a67_4b60_4e98_917d_e178435cfc18.slice/crio-4079dc026fda0c63a8e9790b9f57897db03f448baeeec35d9736b1d3603436e5 WatchSource:0}: Error finding container 4079dc026fda0c63a8e9790b9f57897db03f448baeeec35d9736b1d3603436e5: Status 404 returned error can't find the container with id 4079dc026fda0c63a8e9790b9f57897db03f448baeeec35d9736b1d3603436e5
Jul 07 02:34:16.087672 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:16.087637    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4" event={"ID":"24dd9a67-4b60-4e98-917d-e178435cfc18","Type":"ContainerStarted","Data":"4079dc026fda0c63a8e9790b9f57897db03f448baeeec35d9736b1d3603436e5"}
Jul 07 02:34:18.297147 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:18.297109    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 02:34:18.297554 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:18.297211    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 02:34:18.297554 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:18.297259    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 07 02:34:19.107908 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:19.107872    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4" event={"ID":"24dd9a67-4b60-4e98-917d-e178435cfc18","Type":"ContainerStarted","Data":"ce28424adc45a99fd617bf45e4d3ef18dbab85ba3beaa997136296cdb9f9e97d"}
Jul 07 02:34:19.128999 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:19.128950    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4" podStartSLOduration=2.27227636 podStartE2EDuration="5.128935651s" totalImagesPullingTime="2.856659291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:34:19.127745926 +0000 UTC m=+994.535335615" watchObservedRunningTime="2026-07-07 02:34:19.128935651 +0000 UTC m=+994.536525338"
Jul 07 02:34:19.294192 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:19.294161    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:19.298816 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:19.298784    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:20.113169 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:20.113136    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:20.114188 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:20.114170    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:32.098441 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:32.098402    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"]
Jul 07 02:34:32.099016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:32.098655    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4" podUID="24dd9a67-4b60-4e98-917d-e178435cfc18" containerName="istio-proxy" containerID="cri-o://ce28424adc45a99fd617bf45e4d3ef18dbab85ba3beaa997136296cdb9f9e97d" gracePeriod=30
Jul 07 02:34:36.154571 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:36.154518    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4" podUID="24dd9a67-4b60-4e98-917d-e178435cfc18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 07 02:34:37.487480 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.487446    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"]
Jul 07 02:34:37.491448 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.491428    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.494409 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.494387    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 07 02:34:37.495055 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.495034    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:34:37.495146 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.495107    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:34:37.507073 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.507048    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"]
Jul 07 02:34:37.641314 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.641283    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-dshm\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.641495 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.641322    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfc6q\" (UniqueName: \"kubernetes.io/projected/864e948a-76e7-4135-ab9b-337eaf1aadec-kube-api-access-lfc6q\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.641495 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.641381    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-home\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.641495 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.641414    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.641495 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.641438    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.641495 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.641473    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864e948a-76e7-4135-ab9b-337eaf1aadec-tls-certs\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.641713 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.641576    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-model-cache\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.742930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.742844    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-dshm\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.742930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.742889    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfc6q\" (UniqueName: \"kubernetes.io/projected/864e948a-76e7-4135-ab9b-337eaf1aadec-kube-api-access-lfc6q\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.742930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.742923    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-home\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.743179 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.742955    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.743179 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.742980    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.743179 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.743018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864e948a-76e7-4135-ab9b-337eaf1aadec-tls-certs\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.743179 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.743056    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-model-cache\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.743555 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.743533    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-home\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.743649 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.743629    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.743692 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.743638    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.743759 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.743744    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-model-cache\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.745907 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.745886    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-dshm\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.746169 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.746147    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864e948a-76e7-4135-ab9b-337eaf1aadec-tls-certs\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.751062 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.751036    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfc6q\" (UniqueName: \"kubernetes.io/projected/864e948a-76e7-4135-ab9b-337eaf1aadec-kube-api-access-lfc6q\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.803576 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.803547    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:37.931937 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.923695    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"]
Jul 07 02:34:37.931937 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.926785    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:37.931937 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.929123    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-rwwqb\""
Jul 07 02:34:37.942389 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.942362    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"]
Jul 07 02:34:37.962728 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:37.962707    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"]
Jul 07 02:34:37.966062 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:34:37.966033    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864e948a_76e7_4135_ab9b_337eaf1aadec.slice/crio-0b13225c17119288f13117db519f4c5994d4aa6e28be62aa5f3a77f73a349dc3 WatchSource:0}: Error finding container 0b13225c17119288f13117db519f4c5994d4aa6e28be62aa5f3a77f73a349dc3: Status 404 returned error can't find the container with id 0b13225c17119288f13117db519f4c5994d4aa6e28be62aa5f3a77f73a349dc3
Jul 07 02:34:38.045593 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.045570    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.045700 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.045607    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.045700 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.045624    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.045777 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.045712    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.045777 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.045770    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcnkl\" (UniqueName: \"kubernetes.io/projected/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kube-api-access-jcnkl\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.045845 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.045804    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.147371 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.147319    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.147516 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.147405    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.147516 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.147439    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.147604 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.147542    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.147658 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.147627    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcnkl\" (UniqueName: \"kubernetes.io/projected/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kube-api-access-jcnkl\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.147704 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.147679    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.148061 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.147871    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.148061 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.147922    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.148061 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.147973    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.148061 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.148048    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.150553 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.150532    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.156591 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.156569    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcnkl\" (UniqueName: \"kubernetes.io/projected/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kube-api-access-jcnkl\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.202074 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.201991    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z" event={"ID":"864e948a-76e7-4135-ab9b-337eaf1aadec","Type":"ContainerStarted","Data":"f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af"}
Jul 07 02:34:38.202074 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.202055    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z" event={"ID":"864e948a-76e7-4135-ab9b-337eaf1aadec","Type":"ContainerStarted","Data":"0b13225c17119288f13117db519f4c5994d4aa6e28be62aa5f3a77f73a349dc3"}
Jul 07 02:34:38.203789 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.203767    2570 generic.go:356] "Generic (PLEG): container finished" podID="24dd9a67-4b60-4e98-917d-e178435cfc18" containerID="ce28424adc45a99fd617bf45e4d3ef18dbab85ba3beaa997136296cdb9f9e97d" exitCode=0
Jul 07 02:34:38.203908 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.203850    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4" event={"ID":"24dd9a67-4b60-4e98-917d-e178435cfc18","Type":"ContainerDied","Data":"ce28424adc45a99fd617bf45e4d3ef18dbab85ba3beaa997136296cdb9f9e97d"}
Jul 07 02:34:38.239121 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.239100    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:38.241492 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.241477    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:34:38.349005 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.348971    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-credential-socket\") pod \"24dd9a67-4b60-4e98-917d-e178435cfc18\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") "
Jul 07 02:34:38.349150 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.349041    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-podinfo\") pod \"24dd9a67-4b60-4e98-917d-e178435cfc18\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") "
Jul 07 02:34:38.349150 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.349076    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-data\") pod \"24dd9a67-4b60-4e98-917d-e178435cfc18\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") "
Jul 07 02:34:38.349150 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.349137    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4srj\" (UniqueName: \"kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-kube-api-access-r4srj\") pod \"24dd9a67-4b60-4e98-917d-e178435cfc18\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") "
Jul 07 02:34:38.349321 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.349204    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-certs\") pod \"24dd9a67-4b60-4e98-917d-e178435cfc18\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") "
Jul 07 02:34:38.349321 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.349231    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24dd9a67-4b60-4e98-917d-e178435cfc18-istiod-ca-cert\") pod \"24dd9a67-4b60-4e98-917d-e178435cfc18\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") "
Jul 07 02:34:38.349321 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.349266    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-socket\") pod \"24dd9a67-4b60-4e98-917d-e178435cfc18\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") "
Jul 07 02:34:38.349321 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.349314    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-token\") pod \"24dd9a67-4b60-4e98-917d-e178435cfc18\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") "
Jul 07 02:34:38.349549 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.349387    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-envoy\") pod \"24dd9a67-4b60-4e98-917d-e178435cfc18\" (UID: \"24dd9a67-4b60-4e98-917d-e178435cfc18\") "
Jul 07 02:34:38.350441 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.350195    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-data" (OuterVolumeSpecName: "istio-data") pod "24dd9a67-4b60-4e98-917d-e178435cfc18" (UID: "24dd9a67-4b60-4e98-917d-e178435cfc18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:34:38.350441 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.350286    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24dd9a67-4b60-4e98-917d-e178435cfc18-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "24dd9a67-4b60-4e98-917d-e178435cfc18" (UID: "24dd9a67-4b60-4e98-917d-e178435cfc18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:34:38.350441 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.350401    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "24dd9a67-4b60-4e98-917d-e178435cfc18" (UID: "24dd9a67-4b60-4e98-917d-e178435cfc18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:34:38.350723 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.350455    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "24dd9a67-4b60-4e98-917d-e178435cfc18" (UID: "24dd9a67-4b60-4e98-917d-e178435cfc18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:34:38.351119 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.351083    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "24dd9a67-4b60-4e98-917d-e178435cfc18" (UID: "24dd9a67-4b60-4e98-917d-e178435cfc18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:34:38.352881 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.352840    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "24dd9a67-4b60-4e98-917d-e178435cfc18" (UID: "24dd9a67-4b60-4e98-917d-e178435cfc18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 02:34:38.353033 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.352908    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-kube-api-access-r4srj" (OuterVolumeSpecName: "kube-api-access-r4srj") pod "24dd9a67-4b60-4e98-917d-e178435cfc18" (UID: "24dd9a67-4b60-4e98-917d-e178435cfc18"). InnerVolumeSpecName "kube-api-access-r4srj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:34:38.353110 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.353039    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "24dd9a67-4b60-4e98-917d-e178435cfc18" (UID: "24dd9a67-4b60-4e98-917d-e178435cfc18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:34:38.353230 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.353199    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-token" (OuterVolumeSpecName: "istio-token") pod "24dd9a67-4b60-4e98-917d-e178435cfc18" (UID: "24dd9a67-4b60-4e98-917d-e178435cfc18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:34:38.392603 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.392575    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"]
Jul 07 02:34:38.396126 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:34:38.396102    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ca1e01_c626_4f06_9c53_0bb80ae80219.slice/crio-12c6a5105e7dda038c6ca4417b5af60c25cfe08b5fd0a5cb5de3b51c17f435dd WatchSource:0}: Error finding container 12c6a5105e7dda038c6ca4417b5af60c25cfe08b5fd0a5cb5de3b51c17f435dd: Status 404 returned error can't find the container with id 12c6a5105e7dda038c6ca4417b5af60c25cfe08b5fd0a5cb5de3b51c17f435dd
Jul 07 02:34:38.450439 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.450415    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4srj\" (UniqueName: \"kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-kube-api-access-r4srj\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:34:38.450509 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.450447    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:34:38.450509 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.450465    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24dd9a67-4b60-4e98-917d-e178435cfc18-istiod-ca-cert\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:34:38.450509 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.450478    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-workload-socket\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:34:38.450509 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.450492    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-token\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:34:38.450509 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.450505    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-envoy\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:34:38.450663 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.450517    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-credential-socket\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:34:38.450663 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.450530    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-podinfo\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:34:38.450663 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:38.450542    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24dd9a67-4b60-4e98-917d-e178435cfc18-istio-data\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:34:39.212708 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:39.212663    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"
Jul 07 02:34:39.212708 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:39.212693    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4" event={"ID":"24dd9a67-4b60-4e98-917d-e178435cfc18","Type":"ContainerDied","Data":"4079dc026fda0c63a8e9790b9f57897db03f448baeeec35d9736b1d3603436e5"}
Jul 07 02:34:39.212708 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:39.212768    2570 scope.go:117] "RemoveContainer" containerID="ce28424adc45a99fd617bf45e4d3ef18dbab85ba3beaa997136296cdb9f9e97d"
Jul 07 02:34:39.215842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:39.215717    2570 generic.go:356] "Generic (PLEG): container finished" podID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerID="64265e1ec9ea3de23e50d5ab5af86a2b40b17ef01533fcb5fd6d18f5f647941d" exitCode=0
Jul 07 02:34:39.215987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:39.215843    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" event={"ID":"d8ca1e01-c626-4f06-9c53-0bb80ae80219","Type":"ContainerDied","Data":"64265e1ec9ea3de23e50d5ab5af86a2b40b17ef01533fcb5fd6d18f5f647941d"}
Jul 07 02:34:39.215987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:39.215912    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" event={"ID":"d8ca1e01-c626-4f06-9c53-0bb80ae80219","Type":"ContainerStarted","Data":"12c6a5105e7dda038c6ca4417b5af60c25cfe08b5fd0a5cb5de3b51c17f435dd"}
Jul 07 02:34:39.217845 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:39.217824    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:34:39.234940 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:39.234847    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"]
Jul 07 02:34:39.239825 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:39.239799    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-dlzm4"]
Jul 07 02:34:41.092042 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:41.091876    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24dd9a67-4b60-4e98-917d-e178435cfc18" path="/var/lib/kubelet/pods/24dd9a67-4b60-4e98-917d-e178435cfc18/volumes"
Jul 07 02:34:41.239104 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:41.239016    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" event={"ID":"d8ca1e01-c626-4f06-9c53-0bb80ae80219","Type":"ContainerStarted","Data":"b8e4ac1544c2d32bda1777121935684dab611040087a6798433b39b7813a29fe"}
Jul 07 02:34:43.254504 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:43.254472    2570 generic.go:356] "Generic (PLEG): container finished" podID="864e948a-76e7-4135-ab9b-337eaf1aadec" containerID="f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af" exitCode=0
Jul 07 02:34:43.254961 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:43.254514    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z" event={"ID":"864e948a-76e7-4135-ab9b-337eaf1aadec","Type":"ContainerDied","Data":"f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af"}
Jul 07 02:34:45.267645 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:45.267605    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z" event={"ID":"864e948a-76e7-4135-ab9b-337eaf1aadec","Type":"ContainerStarted","Data":"89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad"}
Jul 07 02:34:45.288437 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:45.288393    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z" podStartSLOduration=2.485824252 podStartE2EDuration="8.288379016s" totalImagesPullingTime="1.159142157s" totalInitContainerRuntime="4.643412607s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:34:45.288032429 +0000 UTC m=+1020.695622116" watchObservedRunningTime="2026-07-07 02:34:45.288379016 +0000 UTC m=+1020.695968703"
Jul 07 02:34:45.804255 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:45.804217    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:47.804604 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:47.804576    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:47.823985 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:47.823964    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:48.299474 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:48.299449    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:34:51.312786 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:51.312752    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg_d8ca1e01-c626-4f06-9c53-0bb80ae80219/main/0.log"
Jul 07 02:34:51.313700 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:51.313660    2570 generic.go:356] "Generic (PLEG): container finished" podID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerID="b8e4ac1544c2d32bda1777121935684dab611040087a6798433b39b7813a29fe" exitCode=1
Jul 07 02:34:51.313814 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:34:51.313704    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" event={"ID":"d8ca1e01-c626-4f06-9c53-0bb80ae80219","Type":"ContainerDied","Data":"b8e4ac1544c2d32bda1777121935684dab611040087a6798433b39b7813a29fe"}
Jul 07 02:35:12.433490 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:12.433458    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg_d8ca1e01-c626-4f06-9c53-0bb80ae80219/main/0.log"
Jul 07 02:35:12.433939 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:12.433917    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" event={"ID":"d8ca1e01-c626-4f06-9c53-0bb80ae80219","Type":"ContainerStarted","Data":"ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306"}
Jul 07 02:35:12.434221 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:12.434205    2570 scope.go:117] "RemoveContainer" containerID="b8e4ac1544c2d32bda1777121935684dab611040087a6798433b39b7813a29fe"
Jul 07 02:35:13.442605 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:13.442578    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg_d8ca1e01-c626-4f06-9c53-0bb80ae80219/main/0.log"
Jul 07 02:35:13.443027 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:13.443006    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" event={"ID":"d8ca1e01-c626-4f06-9c53-0bb80ae80219","Type":"ContainerStarted","Data":"1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6"}
Jul 07 02:35:13.443240 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:13.443226    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:35:13.469470 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:13.469416    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" podStartSLOduration=3.154716556 podStartE2EDuration="36.469394797s" totalImagesPullingTime="32.669871668s" totalInitContainerRuntime="644.806573ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:34:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:35:13.467396671 +0000 UTC m=+1048.874986357" watchObservedRunningTime="2026-07-07 02:35:13.469394797 +0000 UTC m=+1048.876984485"
Jul 07 02:35:18.241986 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:18.241944    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:35:18.242644 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:18.242098    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:35:18.243056 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:35:18.243036    2570 logging.go:55] [core] [Channel #17 SubChannel #18]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.37:9003", ServerName: "10.132.0.37:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.37:9003: connect: connection refused"
Jul 07 02:35:18.243161 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:18.243030    2570 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="tokenizer" probeResult="failure" output="Get \"http://10.132.0.37:8082/healthz\": dial tcp 10.132.0.37:8082: connect: connection refused"
Jul 07 02:35:19.242548 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:19.242493    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.37:9003\" within 1s: context deadline exceeded"
Jul 07 02:35:28.242160 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:35:28.242130    2570 logging.go:55] [core] [Channel #19 SubChannel #20]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.37:9003", ServerName: "10.132.0.37:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.37:9003: connect: connection refused"
Jul 07 02:35:28.243643 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:28.243615    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:35:28.244809 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:28.244788    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:35:29.242284 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:29.242237    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.37:9003\" within 1s: context deadline exceeded"
Jul 07 02:35:48.536924 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:48.536856    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:35:50.215542 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:50.215508    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"]
Jul 07 02:35:50.217876 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:50.217835    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"]
Jul 07 02:35:50.218085 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:50.218053    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="tokenizer" containerID="cri-o://ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306" gracePeriod=60
Jul 07 02:35:50.228596 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:35:50.228551    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:35:50.228596 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:35:50.228596 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:35:50.228596 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6" cmd=["/bin/sleep","15"]
Jul 07 02:35:50.228879 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:50.228623    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" message="" err=<
Jul 07 02:35:50.228879 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:35:50.228879 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:35:50.228879 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:35:50.228879 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:35:50.228639    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:35:50.228879 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:35:50.228879 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:35:50.228879 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="main" containerID="cri-o://1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6"
Jul 07 02:35:50.228879 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:50.228681    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="main" containerID="cri-o://1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6" gracePeriod=60
Jul 07 02:35:50.647697 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:50.647667    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg_d8ca1e01-c626-4f06-9c53-0bb80ae80219/main/0.log"
Jul 07 02:35:50.648109 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:50.648084    2570 generic.go:356] "Generic (PLEG): container finished" podID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerID="1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6" exitCode=0
Jul 07 02:35:50.648211 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:50.648119    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" event={"ID":"d8ca1e01-c626-4f06-9c53-0bb80ae80219","Type":"ContainerDied","Data":"1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6"}
Jul 07 02:35:50.648211 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:50.648171    2570 scope.go:117] "RemoveContainer" containerID="b8e4ac1544c2d32bda1777121935684dab611040087a6798433b39b7813a29fe"
Jul 07 02:35:51.576840 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.576817    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:35:51.631901 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.631878    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-tmp\") pod \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") "
Jul 07 02:35:51.632037 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.631916    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tls-certs\") pod \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") "
Jul 07 02:35:51.632037 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.631934    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-uds\") pod \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") "
Jul 07 02:35:51.632037 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.631955    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kserve-provision-location\") pod \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") "
Jul 07 02:35:51.632037 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.632030    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-cache\") pod \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") "
Jul 07 02:35:51.632276 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.632052    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcnkl\" (UniqueName: \"kubernetes.io/projected/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kube-api-access-jcnkl\") pod \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\" (UID: \"d8ca1e01-c626-4f06-9c53-0bb80ae80219\") "
Jul 07 02:35:51.632276 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.632192    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d8ca1e01-c626-4f06-9c53-0bb80ae80219" (UID: "d8ca1e01-c626-4f06-9c53-0bb80ae80219"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:35:51.632276 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.632209    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "d8ca1e01-c626-4f06-9c53-0bb80ae80219" (UID: "d8ca1e01-c626-4f06-9c53-0bb80ae80219"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:35:51.632469 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.632441    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d8ca1e01-c626-4f06-9c53-0bb80ae80219" (UID: "d8ca1e01-c626-4f06-9c53-0bb80ae80219"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:35:51.632589 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.632446    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-tmp\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:35:51.632589 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.632495    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:35:51.632759 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.632736    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d8ca1e01-c626-4f06-9c53-0bb80ae80219" (UID: "d8ca1e01-c626-4f06-9c53-0bb80ae80219"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:35:51.634136 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.634118    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8ca1e01-c626-4f06-9c53-0bb80ae80219" (UID: "d8ca1e01-c626-4f06-9c53-0bb80ae80219"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:35:51.634262 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.634240    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kube-api-access-jcnkl" (OuterVolumeSpecName: "kube-api-access-jcnkl") pod "d8ca1e01-c626-4f06-9c53-0bb80ae80219" (UID: "d8ca1e01-c626-4f06-9c53-0bb80ae80219"). InnerVolumeSpecName "kube-api-access-jcnkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:35:51.654237 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.654212    2570 generic.go:356] "Generic (PLEG): container finished" podID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerID="ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306" exitCode=0
Jul 07 02:35:51.654322 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.654259    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" event={"ID":"d8ca1e01-c626-4f06-9c53-0bb80ae80219","Type":"ContainerDied","Data":"ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306"}
Jul 07 02:35:51.654322 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.654266    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"
Jul 07 02:35:51.654424 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.654319    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg" event={"ID":"d8ca1e01-c626-4f06-9c53-0bb80ae80219","Type":"ContainerDied","Data":"12c6a5105e7dda038c6ca4417b5af60c25cfe08b5fd0a5cb5de3b51c17f435dd"}
Jul 07 02:35:51.654424 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.654374    2570 scope.go:117] "RemoveContainer" containerID="1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6"
Jul 07 02:35:51.663489 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.663469    2570 scope.go:117] "RemoveContainer" containerID="ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306"
Jul 07 02:35:51.671064 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.671049    2570 scope.go:117] "RemoveContainer" containerID="64265e1ec9ea3de23e50d5ab5af86a2b40b17ef01533fcb5fd6d18f5f647941d"
Jul 07 02:35:51.674850 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.674829    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"]
Jul 07 02:35:51.677919 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.677900    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5f5d4465g6nfg"]
Jul 07 02:35:51.679270 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.679256    2570 scope.go:117] "RemoveContainer" containerID="1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6"
Jul 07 02:35:51.679583 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:35:51.679563    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6\": container with ID starting with 1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6 not found: ID does not exist" containerID="1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6"
Jul 07 02:35:51.679649 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.679592    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6"} err="failed to get container status \"1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6\": rpc error: code = NotFound desc = could not find container \"1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6\": container with ID starting with 1e25bf2911c17c6113fd85ff778168b6b1f8f6d49c4b0dabcfdbcfd8fc1ed5a6 not found: ID does not exist"
Jul 07 02:35:51.679649 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.679608    2570 scope.go:117] "RemoveContainer" containerID="ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306"
Jul 07 02:35:51.679830 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:35:51.679813    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306\": container with ID starting with ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306 not found: ID does not exist" containerID="ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306"
Jul 07 02:35:51.679870 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.679835    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306"} err="failed to get container status \"ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306\": rpc error: code = NotFound desc = could not find container \"ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306\": container with ID starting with ea297320c78b442778937b9a20a61db10b0b825971e9c7c5e83dbe79713b6306 not found: ID does not exist"
Jul 07 02:35:51.679870 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.679850    2570 scope.go:117] "RemoveContainer" containerID="64265e1ec9ea3de23e50d5ab5af86a2b40b17ef01533fcb5fd6d18f5f647941d"
Jul 07 02:35:51.680042 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:35:51.680025    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64265e1ec9ea3de23e50d5ab5af86a2b40b17ef01533fcb5fd6d18f5f647941d\": container with ID starting with 64265e1ec9ea3de23e50d5ab5af86a2b40b17ef01533fcb5fd6d18f5f647941d not found: ID does not exist" containerID="64265e1ec9ea3de23e50d5ab5af86a2b40b17ef01533fcb5fd6d18f5f647941d"
Jul 07 02:35:51.680082 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.680046    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64265e1ec9ea3de23e50d5ab5af86a2b40b17ef01533fcb5fd6d18f5f647941d"} err="failed to get container status \"64265e1ec9ea3de23e50d5ab5af86a2b40b17ef01533fcb5fd6d18f5f647941d\": rpc error: code = NotFound desc = could not find container \"64265e1ec9ea3de23e50d5ab5af86a2b40b17ef01533fcb5fd6d18f5f647941d\": container with ID starting with 64265e1ec9ea3de23e50d5ab5af86a2b40b17ef01533fcb5fd6d18f5f647941d not found: ID does not exist"
Jul 07 02:35:51.733810 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.733791    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:35:51.733810 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.733812    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:35:51.733919 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.733822    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8ca1e01-c626-4f06-9c53-0bb80ae80219-tokenizer-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:35:51.733919 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:51.733831    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcnkl\" (UniqueName: \"kubernetes.io/projected/d8ca1e01-c626-4f06-9c53-0bb80ae80219-kube-api-access-jcnkl\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:35:53.087415 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:35:53.087384    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" path="/var/lib/kubelet/pods/d8ca1e01-c626-4f06-9c53-0bb80ae80219/volumes"
Jul 07 02:36:05.245046 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.245006    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z" podUID="864e948a-76e7-4135-ab9b-337eaf1aadec" containerName="main" containerID="cri-o://89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad" gracePeriod=45
Jul 07 02:36:05.287725 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.287689    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z" podUID="864e948a-76e7-4135-ab9b-337eaf1aadec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 07 02:36:05.522387 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.522339    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:36:05.644130 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644100    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-model-cache\") pod \"864e948a-76e7-4135-ab9b-337eaf1aadec\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") "
Jul 07 02:36:05.644280 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644157    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfc6q\" (UniqueName: \"kubernetes.io/projected/864e948a-76e7-4135-ab9b-337eaf1aadec-kube-api-access-lfc6q\") pod \"864e948a-76e7-4135-ab9b-337eaf1aadec\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") "
Jul 07 02:36:05.644280 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644205    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-dshm\") pod \"864e948a-76e7-4135-ab9b-337eaf1aadec\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") "
Jul 07 02:36:05.644280 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644221    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864e948a-76e7-4135-ab9b-337eaf1aadec-tls-certs\") pod \"864e948a-76e7-4135-ab9b-337eaf1aadec\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") "
Jul 07 02:36:05.644280 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644248    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-kserve-provision-location\") pod \"864e948a-76e7-4135-ab9b-337eaf1aadec\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") "
Jul 07 02:36:05.644518 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644306    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-home\") pod \"864e948a-76e7-4135-ab9b-337eaf1aadec\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") "
Jul 07 02:36:05.644518 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644339    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-model-cache" (OuterVolumeSpecName: "model-cache") pod "864e948a-76e7-4135-ab9b-337eaf1aadec" (UID: "864e948a-76e7-4135-ab9b-337eaf1aadec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:05.644518 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644381    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-tmp-dir\") pod \"864e948a-76e7-4135-ab9b-337eaf1aadec\" (UID: \"864e948a-76e7-4135-ab9b-337eaf1aadec\") "
Jul 07 02:36:05.644657 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644602    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "864e948a-76e7-4135-ab9b-337eaf1aadec" (UID: "864e948a-76e7-4135-ab9b-337eaf1aadec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:05.644657 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644620    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-home" (OuterVolumeSpecName: "home") pod "864e948a-76e7-4135-ab9b-337eaf1aadec" (UID: "864e948a-76e7-4135-ab9b-337eaf1aadec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:05.644765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644729    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-home\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:36:05.644765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644750    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-tmp-dir\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:36:05.644845 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.644765    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:36:05.646587 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.646553    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864e948a-76e7-4135-ab9b-337eaf1aadec-kube-api-access-lfc6q" (OuterVolumeSpecName: "kube-api-access-lfc6q") pod "864e948a-76e7-4135-ab9b-337eaf1aadec" (UID: "864e948a-76e7-4135-ab9b-337eaf1aadec"). InnerVolumeSpecName "kube-api-access-lfc6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:36:05.647015 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.646996    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-dshm" (OuterVolumeSpecName: "dshm") pod "864e948a-76e7-4135-ab9b-337eaf1aadec" (UID: "864e948a-76e7-4135-ab9b-337eaf1aadec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:05.647147 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.647130    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864e948a-76e7-4135-ab9b-337eaf1aadec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "864e948a-76e7-4135-ab9b-337eaf1aadec" (UID: "864e948a-76e7-4135-ab9b-337eaf1aadec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:36:05.699842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.699806    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "864e948a-76e7-4135-ab9b-337eaf1aadec" (UID: "864e948a-76e7-4135-ab9b-337eaf1aadec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:05.723756 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.723724    2570 generic.go:356] "Generic (PLEG): container finished" podID="864e948a-76e7-4135-ab9b-337eaf1aadec" containerID="89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad" exitCode=0
Jul 07 02:36:05.723886 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.723775    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"
Jul 07 02:36:05.723886 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.723815    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z" event={"ID":"864e948a-76e7-4135-ab9b-337eaf1aadec","Type":"ContainerDied","Data":"89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad"}
Jul 07 02:36:05.723886 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.723872    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z" event={"ID":"864e948a-76e7-4135-ab9b-337eaf1aadec","Type":"ContainerDied","Data":"0b13225c17119288f13117db519f4c5994d4aa6e28be62aa5f3a77f73a349dc3"}
Jul 07 02:36:05.724000 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.723895    2570 scope.go:117] "RemoveContainer" containerID="89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad"
Jul 07 02:36:05.732813 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.732795    2570 scope.go:117] "RemoveContainer" containerID="f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af"
Jul 07 02:36:05.740679 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.740663    2570 scope.go:117] "RemoveContainer" containerID="89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad"
Jul 07 02:36:05.740891 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:36:05.740874    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad\": container with ID starting with 89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad not found: ID does not exist" containerID="89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad"
Jul 07 02:36:05.740946 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.740899    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad"} err="failed to get container status \"89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad\": rpc error: code = NotFound desc = could not find container \"89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad\": container with ID starting with 89203e33ab5459a148d482999cba1f3d1fc70ecd860bb84191f6fd78355da0ad not found: ID does not exist"
Jul 07 02:36:05.740946 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.740914    2570 scope.go:117] "RemoveContainer" containerID="f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af"
Jul 07 02:36:05.741116 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:36:05.741096    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af\": container with ID starting with f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af not found: ID does not exist" containerID="f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af"
Jul 07 02:36:05.741161 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.741123    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af"} err="failed to get container status \"f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af\": rpc error: code = NotFound desc = could not find container \"f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af\": container with ID starting with f28c1d40b6e3ae4fc29ab66c4239638964734e4a74c4c972f009ca36c02807af not found: ID does not exist"
Jul 07 02:36:05.745160 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.745133    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfc6q\" (UniqueName: \"kubernetes.io/projected/864e948a-76e7-4135-ab9b-337eaf1aadec-kube-api-access-lfc6q\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:36:05.745242 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.745163    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-dshm\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:36:05.745242 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.745173    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/864e948a-76e7-4135-ab9b-337eaf1aadec-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:36:05.745242 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.745182    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/864e948a-76e7-4135-ab9b-337eaf1aadec-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:36:05.745762 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.745743    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"]
Jul 07 02:36:05.749460 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:05.749440    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-xxw4z"]
Jul 07 02:36:07.093848 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:07.093817    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="864e948a-76e7-4135-ab9b-337eaf1aadec" path="/var/lib/kubelet/pods/864e948a-76e7-4135-ab9b-337eaf1aadec/volumes"
Jul 07 02:36:21.463782 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.463745    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"]
Jul 07 02:36:21.464339 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464276    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864e948a-76e7-4135-ab9b-337eaf1aadec" containerName="main"
Jul 07 02:36:21.464339 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464295    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="864e948a-76e7-4135-ab9b-337eaf1aadec" containerName="main"
Jul 07 02:36:21.464339 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464314    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864e948a-76e7-4135-ab9b-337eaf1aadec" containerName="storage-initializer"
Jul 07 02:36:21.464339 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464322    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="864e948a-76e7-4135-ab9b-337eaf1aadec" containerName="storage-initializer"
Jul 07 02:36:21.464339 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464364    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="tokenizer"
Jul 07 02:36:21.464339 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464373    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="tokenizer"
Jul 07 02:36:21.464693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464385    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="storage-initializer"
Jul 07 02:36:21.464693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464393    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="storage-initializer"
Jul 07 02:36:21.464693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464407    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24dd9a67-4b60-4e98-917d-e178435cfc18" containerName="istio-proxy"
Jul 07 02:36:21.464693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464414    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="24dd9a67-4b60-4e98-917d-e178435cfc18" containerName="istio-proxy"
Jul 07 02:36:21.464693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464424    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="main"
Jul 07 02:36:21.464693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464431    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="main"
Jul 07 02:36:21.464693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464550    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="24dd9a67-4b60-4e98-917d-e178435cfc18" containerName="istio-proxy"
Jul 07 02:36:21.464693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464565    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="main"
Jul 07 02:36:21.464693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464576    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="864e948a-76e7-4135-ab9b-337eaf1aadec" containerName="main"
Jul 07 02:36:21.464693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464587    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="main"
Jul 07 02:36:21.464693 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.464595    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="tokenizer"
Jul 07 02:36:21.467786 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.467767    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:21.471592 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.471569    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:36:21.471725 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.471627    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 07 02:36:21.471798 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.471742    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:36:21.471860 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.471799    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-6tdrz\""
Jul 07 02:36:21.471860 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.471850    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:36:21.481502 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.481479    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"]
Jul 07 02:36:21.573027 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.572992    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e837c028-0531-4427-a3bb-3777ed85f6a6-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:21.573197 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.573169    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxcg\" (UniqueName: \"kubernetes.io/projected/e837c028-0531-4427-a3bb-3777ed85f6a6-kube-api-access-7wxcg\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:21.573289 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.573267    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e837c028-0531-4427-a3bb-3777ed85f6a6-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:21.674660 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.674627    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e837c028-0531-4427-a3bb-3777ed85f6a6-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:21.674817 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.674686    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxcg\" (UniqueName: \"kubernetes.io/projected/e837c028-0531-4427-a3bb-3777ed85f6a6-kube-api-access-7wxcg\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:21.674817 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.674744    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e837c028-0531-4427-a3bb-3777ed85f6a6-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:21.675196 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.675175    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e837c028-0531-4427-a3bb-3777ed85f6a6-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:21.677185 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.677161    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e837c028-0531-4427-a3bb-3777ed85f6a6-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:21.684032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.683998    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxcg\" (UniqueName: \"kubernetes.io/projected/e837c028-0531-4427-a3bb-3777ed85f6a6-kube-api-access-7wxcg\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:21.778481 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.778453    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:21.910814 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:21.910787    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"]
Jul 07 02:36:21.913193 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:36:21.913166    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode837c028_0531_4427_a3bb_3777ed85f6a6.slice/crio-b88a3e43c955900bd14a156a730fd39d6c4c0cbfc4a215cad5d520e4e69d2cf6 WatchSource:0}: Error finding container b88a3e43c955900bd14a156a730fd39d6c4c0cbfc4a215cad5d520e4e69d2cf6: Status 404 returned error can't find the container with id b88a3e43c955900bd14a156a730fd39d6c4c0cbfc4a215cad5d520e4e69d2cf6
Jul 07 02:36:22.813556 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:22.813523    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7" event={"ID":"e837c028-0531-4427-a3bb-3777ed85f6a6","Type":"ContainerStarted","Data":"2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9"}
Jul 07 02:36:22.813902 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:22.813573    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7" event={"ID":"e837c028-0531-4427-a3bb-3777ed85f6a6","Type":"ContainerStarted","Data":"b88a3e43c955900bd14a156a730fd39d6c4c0cbfc4a215cad5d520e4e69d2cf6"}
Jul 07 02:36:22.813902 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:22.813594    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:22.829751 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:22.829709    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7" podStartSLOduration=1.829686521 podStartE2EDuration="1.829686521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:36:22.828281617 +0000 UTC m=+1118.235871305" watchObservedRunningTime="2026-07-07 02:36:22.829686521 +0000 UTC m=+1118.237276207"
Jul 07 02:36:53.820866 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:53.820835    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:55.572767 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.572735    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"]
Jul 07 02:36:55.582369 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:36:55.582308    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:36:55.582369 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:36:55.582369 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:36:55.582369 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9" cmd=["/bin/sleep","15"]
Jul 07 02:36:55.582631 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.582392    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7" message="" err=<
Jul 07 02:36:55.582631 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:36:55.582631 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:36:55.582631 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:36:55.582631 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:36:55.582412    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:36:55.582631 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:36:55.582631 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:36:55.582631 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7" podUID="e837c028-0531-4427-a3bb-3777ed85f6a6" containerName="main" containerID="cri-o://2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9"
Jul 07 02:36:55.582631 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.582455    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7" podUID="e837c028-0531-4427-a3bb-3777ed85f6a6" containerName="main" containerID="cri-o://2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9" gracePeriod=60
Jul 07 02:36:55.833296 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.833229    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:55.950454 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.950426    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e837c028-0531-4427-a3bb-3777ed85f6a6-tokenizer-uds\") pod \"e837c028-0531-4427-a3bb-3777ed85f6a6\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") "
Jul 07 02:36:55.950593 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.950461    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wxcg\" (UniqueName: \"kubernetes.io/projected/e837c028-0531-4427-a3bb-3777ed85f6a6-kube-api-access-7wxcg\") pod \"e837c028-0531-4427-a3bb-3777ed85f6a6\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") "
Jul 07 02:36:55.950593 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.950493    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e837c028-0531-4427-a3bb-3777ed85f6a6-tls-certs\") pod \"e837c028-0531-4427-a3bb-3777ed85f6a6\" (UID: \"e837c028-0531-4427-a3bb-3777ed85f6a6\") "
Jul 07 02:36:55.950755 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.950725    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e837c028-0531-4427-a3bb-3777ed85f6a6-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e837c028-0531-4427-a3bb-3777ed85f6a6" (UID: "e837c028-0531-4427-a3bb-3777ed85f6a6"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:55.952685 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.952664    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e837c028-0531-4427-a3bb-3777ed85f6a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e837c028-0531-4427-a3bb-3777ed85f6a6" (UID: "e837c028-0531-4427-a3bb-3777ed85f6a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:36:55.952792 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.952708    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e837c028-0531-4427-a3bb-3777ed85f6a6-kube-api-access-7wxcg" (OuterVolumeSpecName: "kube-api-access-7wxcg") pod "e837c028-0531-4427-a3bb-3777ed85f6a6" (UID: "e837c028-0531-4427-a3bb-3777ed85f6a6"). InnerVolumeSpecName "kube-api-access-7wxcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:36:55.980491 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.980462    2570 generic.go:356] "Generic (PLEG): container finished" podID="e837c028-0531-4427-a3bb-3777ed85f6a6" containerID="2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9" exitCode=0
Jul 07 02:36:55.980606 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.980504    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"
Jul 07 02:36:55.980606 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.980544    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7" event={"ID":"e837c028-0531-4427-a3bb-3777ed85f6a6","Type":"ContainerDied","Data":"2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9"}
Jul 07 02:36:55.980606 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.980592    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7" event={"ID":"e837c028-0531-4427-a3bb-3777ed85f6a6","Type":"ContainerDied","Data":"b88a3e43c955900bd14a156a730fd39d6c4c0cbfc4a215cad5d520e4e69d2cf6"}
Jul 07 02:36:55.980724 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.980611    2570 scope.go:117] "RemoveContainer" containerID="2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9"
Jul 07 02:36:55.989085 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.989071    2570 scope.go:117] "RemoveContainer" containerID="2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9"
Jul 07 02:36:55.989320 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:36:55.989305    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9\": container with ID starting with 2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9 not found: ID does not exist" containerID="2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9"
Jul 07 02:36:55.989407 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:55.989327    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9"} err="failed to get container status \"2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9\": rpc error: code = NotFound desc = could not find container \"2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9\": container with ID starting with 2505247979ab9377f39006af62f5b1c872df715a86757b6735eddb90c06bf4b9 not found: ID does not exist"
Jul 07 02:36:56.002645 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:56.002625    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"]
Jul 07 02:36:56.011429 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:56.011409    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scherhlm7"]
Jul 07 02:36:56.051054 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:56.051031    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e837c028-0531-4427-a3bb-3777ed85f6a6-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:36:56.051054 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:56.051050    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wxcg\" (UniqueName: \"kubernetes.io/projected/e837c028-0531-4427-a3bb-3777ed85f6a6-kube-api-access-7wxcg\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:36:56.051174 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:56.051060    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e837c028-0531-4427-a3bb-3777ed85f6a6-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:36:57.086518 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:36:57.086486    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e837c028-0531-4427-a3bb-3777ed85f6a6" path="/var/lib/kubelet/pods/e837c028-0531-4427-a3bb-3777ed85f6a6/volumes"
Jul 07 02:37:26.455729 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.455695    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"]
Jul 07 02:37:26.456287 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.456272    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="main"
Jul 07 02:37:26.456334 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.456292    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ca1e01-c626-4f06-9c53-0bb80ae80219" containerName="main"
Jul 07 02:37:26.456334 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.456308    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e837c028-0531-4427-a3bb-3777ed85f6a6" containerName="main"
Jul 07 02:37:26.456334 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.456316    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e837c028-0531-4427-a3bb-3777ed85f6a6" containerName="main"
Jul 07 02:37:26.456520 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.456442    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e837c028-0531-4427-a3bb-3777ed85f6a6" containerName="main"
Jul 07 02:37:26.459857 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.459837    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:26.463118 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.463098    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-t7jf2\""
Jul 07 02:37:26.463229 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.463114    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:37:26.463229 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.463136    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:37:26.463229 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.463163    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:37:26.463229 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.463214    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 07 02:37:26.469498 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.469478    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"]
Jul 07 02:37:26.593314 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.593285    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkrq5\" (UniqueName: \"kubernetes.io/projected/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-kube-api-access-mkrq5\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:26.593438 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.593317    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:26.593438 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.593396    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:26.694317 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.694288    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkrq5\" (UniqueName: \"kubernetes.io/projected/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-kube-api-access-mkrq5\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:26.694454 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.694321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:26.694454 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.694400    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:26.694796 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.694776    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:26.696901 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.696884    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:26.701929 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.701907    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkrq5\" (UniqueName: \"kubernetes.io/projected/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-kube-api-access-mkrq5\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:26.770555 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.770486    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:26.904381 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:26.904329    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"]
Jul 07 02:37:26.906151 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:37:26.906125    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acb7cf4_1b7e_4ed6_b0ad_6aa0bd10c385.slice/crio-3111fcb0113b7cad1011ea3bcc9ba15c722cd06781cd037dd5680c0f8a44d812 WatchSource:0}: Error finding container 3111fcb0113b7cad1011ea3bcc9ba15c722cd06781cd037dd5680c0f8a44d812: Status 404 returned error can't find the container with id 3111fcb0113b7cad1011ea3bcc9ba15c722cd06781cd037dd5680c0f8a44d812
Jul 07 02:37:27.126013 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:27.125977    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd" event={"ID":"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385","Type":"ContainerStarted","Data":"d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274"}
Jul 07 02:37:27.126195 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:27.126028    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd" event={"ID":"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385","Type":"ContainerStarted","Data":"3111fcb0113b7cad1011ea3bcc9ba15c722cd06781cd037dd5680c0f8a44d812"}
Jul 07 02:37:27.126195 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:27.126119    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:37:27.149419 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:27.149370    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd" podStartSLOduration=1.149339889 podStartE2EDuration="1.149339889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:37:27.147590857 +0000 UTC m=+1182.555180554" watchObservedRunningTime="2026-07-07 02:37:27.149339889 +0000 UTC m=+1182.556929574"
Jul 07 02:37:58.133566 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:37:58.133530    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:38:33.574220 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.574188    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"]
Jul 07 02:38:33.577789 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.577771    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:33.580823 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.580803    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-hfsj9\""
Jul 07 02:38:33.581660 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.581643    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 07 02:38:33.593146 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.593124    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"]
Jul 07 02:38:33.734765 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.734730    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1be9541e-e063-468f-aff4-bd8ffc57098d-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:33.734876 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.734788    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1be9541e-e063-468f-aff4-bd8ffc57098d-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:33.734876 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.734855    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw7md\" (UniqueName: \"kubernetes.io/projected/1be9541e-e063-468f-aff4-bd8ffc57098d-kube-api-access-cw7md\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:33.835332 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.835266    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1be9541e-e063-468f-aff4-bd8ffc57098d-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:33.835332 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.835304    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1be9541e-e063-468f-aff4-bd8ffc57098d-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:33.835332 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.835329    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7md\" (UniqueName: \"kubernetes.io/projected/1be9541e-e063-468f-aff4-bd8ffc57098d-kube-api-access-cw7md\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:33.835722 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.835701    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1be9541e-e063-468f-aff4-bd8ffc57098d-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:33.837783 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.837766    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1be9541e-e063-468f-aff4-bd8ffc57098d-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:33.845663 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.845640    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7md\" (UniqueName: \"kubernetes.io/projected/1be9541e-e063-468f-aff4-bd8ffc57098d-kube-api-access-cw7md\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:33.887090 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:33.887063    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:34.020742 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:34.020659    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"]
Jul 07 02:38:34.023417 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:38:34.023391    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be9541e_e063_468f_aff4_bd8ffc57098d.slice/crio-3f12d994c160e184e395369444612d105da7d60e90738f0942dfc2b03227af91 WatchSource:0}: Error finding container 3f12d994c160e184e395369444612d105da7d60e90738f0942dfc2b03227af91: Status 404 returned error can't find the container with id 3f12d994c160e184e395369444612d105da7d60e90738f0942dfc2b03227af91
Jul 07 02:38:34.447582 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:34.447552    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf" event={"ID":"1be9541e-e063-468f-aff4-bd8ffc57098d","Type":"ContainerStarted","Data":"dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8"}
Jul 07 02:38:34.447776 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:34.447600    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf" event={"ID":"1be9541e-e063-468f-aff4-bd8ffc57098d","Type":"ContainerStarted","Data":"3f12d994c160e184e395369444612d105da7d60e90738f0942dfc2b03227af91"}
Jul 07 02:38:34.447776 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:34.447634    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:38:34.468189 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:38:34.468147    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf" podStartSLOduration=1.468131931 podStartE2EDuration="1.468131931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:38:34.466735588 +0000 UTC m=+1249.874325276" watchObservedRunningTime="2026-07-07 02:38:34.468131931 +0000 UTC m=+1249.875721617"
Jul 07 02:39:05.453784 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:05.453714    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:39:27.690818 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:27.690783    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"]
Jul 07 02:39:27.700493 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:39:27.700456    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:39:27.700493 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:39:27.700493 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:39:27.700493 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8" cmd=["/bin/sleep","15"]
Jul 07 02:39:27.700781 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:27.700503    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf" message="" err=<
Jul 07 02:39:27.700781 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:39:27.700781 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:39:27.700781 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:39:27.700781 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:39:27.700516    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:39:27.700781 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:39:27.700781 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:39:27.700781 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf" podUID="1be9541e-e063-468f-aff4-bd8ffc57098d" containerName="main" containerID="cri-o://dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8"
Jul 07 02:39:27.700781 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:27.700561    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf" podUID="1be9541e-e063-468f-aff4-bd8ffc57098d" containerName="main" containerID="cri-o://dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8" gracePeriod=60
Jul 07 02:39:27.950068 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:27.950015    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:39:28.078319 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.078292    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1be9541e-e063-468f-aff4-bd8ffc57098d-tokenizer-uds\") pod \"1be9541e-e063-468f-aff4-bd8ffc57098d\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") "
Jul 07 02:39:28.078319 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.078331    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1be9541e-e063-468f-aff4-bd8ffc57098d-tls-certs\") pod \"1be9541e-e063-468f-aff4-bd8ffc57098d\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") "
Jul 07 02:39:28.078590 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.078436    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw7md\" (UniqueName: \"kubernetes.io/projected/1be9541e-e063-468f-aff4-bd8ffc57098d-kube-api-access-cw7md\") pod \"1be9541e-e063-468f-aff4-bd8ffc57098d\" (UID: \"1be9541e-e063-468f-aff4-bd8ffc57098d\") "
Jul 07 02:39:28.078647 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.078623    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be9541e-e063-468f-aff4-bd8ffc57098d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1be9541e-e063-468f-aff4-bd8ffc57098d" (UID: "1be9541e-e063-468f-aff4-bd8ffc57098d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:39:28.080612 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.080591    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1be9541e-e063-468f-aff4-bd8ffc57098d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1be9541e-e063-468f-aff4-bd8ffc57098d" (UID: "1be9541e-e063-468f-aff4-bd8ffc57098d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:39:28.080757 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.080732    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be9541e-e063-468f-aff4-bd8ffc57098d-kube-api-access-cw7md" (OuterVolumeSpecName: "kube-api-access-cw7md") pod "1be9541e-e063-468f-aff4-bd8ffc57098d" (UID: "1be9541e-e063-468f-aff4-bd8ffc57098d"). InnerVolumeSpecName "kube-api-access-cw7md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:39:28.179156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.179137    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1be9541e-e063-468f-aff4-bd8ffc57098d-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:39:28.179156 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.179157    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1be9541e-e063-468f-aff4-bd8ffc57098d-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:39:28.179296 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.179166    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw7md\" (UniqueName: \"kubernetes.io/projected/1be9541e-e063-468f-aff4-bd8ffc57098d-kube-api-access-cw7md\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:39:28.712773 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.712740    2570 generic.go:356] "Generic (PLEG): container finished" podID="1be9541e-e063-468f-aff4-bd8ffc57098d" containerID="dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8" exitCode=0
Jul 07 02:39:28.713133 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.712786    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"
Jul 07 02:39:28.713133 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.712822    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf" event={"ID":"1be9541e-e063-468f-aff4-bd8ffc57098d","Type":"ContainerDied","Data":"dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8"}
Jul 07 02:39:28.713133 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.712876    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf" event={"ID":"1be9541e-e063-468f-aff4-bd8ffc57098d","Type":"ContainerDied","Data":"3f12d994c160e184e395369444612d105da7d60e90738f0942dfc2b03227af91"}
Jul 07 02:39:28.713133 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.712895    2570 scope.go:117] "RemoveContainer" containerID="dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8"
Jul 07 02:39:28.722015 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.722000    2570 scope.go:117] "RemoveContainer" containerID="dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8"
Jul 07 02:39:28.722268 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:39:28.722247    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8\": container with ID starting with dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8 not found: ID does not exist" containerID="dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8"
Jul 07 02:39:28.722462 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.722273    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8"} err="failed to get container status \"dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8\": rpc error: code = NotFound desc = could not find container \"dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8\": container with ID starting with dbd5ebfb1a55d614e3b148d86168780521a83b9735281a984d3677845ec020a8 not found: ID does not exist"
Jul 07 02:39:28.735659 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.735637    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"]
Jul 07 02:39:28.739677 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:28.739658    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scherh7zf"]
Jul 07 02:39:29.087045 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:29.087016    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1be9541e-e063-468f-aff4-bd8ffc57098d" path="/var/lib/kubelet/pods/1be9541e-e063-468f-aff4-bd8ffc57098d/volumes"
Jul 07 02:39:47.493057 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.493022    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"]
Jul 07 02:39:47.503536 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:39:47.503495    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:39:47.503536 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:39:47.503536 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:39:47.503536 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274" cmd=["/bin/sleep","15"]
Jul 07 02:39:47.503856 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.503555    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd" message="" err=<
Jul 07 02:39:47.503856 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:39:47.503856 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:39:47.503856 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:39:47.503856 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:39:47.503572    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:39:47.503856 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:39:47.503856 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:39:47.503856 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd" podUID="1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385" containerName="main" containerID="cri-o://d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274"
Jul 07 02:39:47.503856 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.503631    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd" podUID="1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385" containerName="main" containerID="cri-o://d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274" gracePeriod=60
Jul 07 02:39:47.764023 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.763995    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:39:47.814264 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.814233    2570 generic.go:356] "Generic (PLEG): container finished" podID="1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385" containerID="d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274" exitCode=0
Jul 07 02:39:47.814424 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.814283    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"
Jul 07 02:39:47.814424 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.814313    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd" event={"ID":"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385","Type":"ContainerDied","Data":"d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274"}
Jul 07 02:39:47.814424 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.814387    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd" event={"ID":"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385","Type":"ContainerDied","Data":"3111fcb0113b7cad1011ea3bcc9ba15c722cd06781cd037dd5680c0f8a44d812"}
Jul 07 02:39:47.814424 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.814408    2570 scope.go:117] "RemoveContainer" containerID="d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274"
Jul 07 02:39:47.822649 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.822633    2570 scope.go:117] "RemoveContainer" containerID="d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274"
Jul 07 02:39:47.822898 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:39:47.822880    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274\": container with ID starting with d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274 not found: ID does not exist" containerID="d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274"
Jul 07 02:39:47.822951 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.822907    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274"} err="failed to get container status \"d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274\": rpc error: code = NotFound desc = could not find container \"d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274\": container with ID starting with d4198ffffd78a6e6af206f0b2284fa4a3b65a260b583023201d7eeac580fb274 not found: ID does not exist"
Jul 07 02:39:47.830186 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.830168    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkrq5\" (UniqueName: \"kubernetes.io/projected/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-kube-api-access-mkrq5\") pod \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") "
Jul 07 02:39:47.830268 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.830244    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tls-certs\") pod \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") "
Jul 07 02:39:47.830315 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.830278    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tokenizer-uds\") pod \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\" (UID: \"1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385\") "
Jul 07 02:39:47.830563 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.830544    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385" (UID: "1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:39:47.832288 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.832268    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385" (UID: "1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:39:47.832408 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.832334    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-kube-api-access-mkrq5" (OuterVolumeSpecName: "kube-api-access-mkrq5") pod "1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385" (UID: "1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385"). InnerVolumeSpecName "kube-api-access-mkrq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:39:47.931534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.931513    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkrq5\" (UniqueName: \"kubernetes.io/projected/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-kube-api-access-mkrq5\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:39:47.931534 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.931533    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:39:47.931654 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:47.931542    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:39:48.136259 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:48.136233    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"]
Jul 07 02:39:48.141727 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:48.141708    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekjmqd"]
Jul 07 02:39:49.086588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:39:49.086552    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385" path="/var/lib/kubelet/pods/1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385/volumes"
Jul 07 02:40:22.480338 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.480305    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"]
Jul 07 02:40:22.480763 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.480703    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385" containerName="main"
Jul 07 02:40:22.480763 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.480715    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385" containerName="main"
Jul 07 02:40:22.480763 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.480734    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1be9541e-e063-468f-aff4-bd8ffc57098d" containerName="main"
Jul 07 02:40:22.480763 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.480740    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1be9541e-e063-468f-aff4-bd8ffc57098d" containerName="main"
Jul 07 02:40:22.480891 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.480792    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1be9541e-e063-468f-aff4-bd8ffc57098d" containerName="main"
Jul 07 02:40:22.480891 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.480804    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1acb7cf4-1b7e-4ed6-b0ad-6aa0bd10c385" containerName="main"
Jul 07 02:40:22.483944 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.483928    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:22.487764 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.487744    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 07 02:40:22.487890 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.487782    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-65fvk\""
Jul 07 02:40:22.487967 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.487951    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:40:22.488023 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.487974    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:40:22.488023 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.488012    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:40:22.495201 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.495180    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"]
Jul 07 02:40:22.596901 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.596875    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660875c4-4375-4280-8ccb-2957456b68df-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:22.597035 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.596926    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/660875c4-4375-4280-8ccb-2957456b68df-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:22.597035 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.596997    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbbfp\" (UniqueName: \"kubernetes.io/projected/660875c4-4375-4280-8ccb-2957456b68df-kube-api-access-sbbfp\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:22.697916 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.697887    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660875c4-4375-4280-8ccb-2957456b68df-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:22.698051 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.697941    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/660875c4-4375-4280-8ccb-2957456b68df-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:22.698051 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.697972    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbbfp\" (UniqueName: \"kubernetes.io/projected/660875c4-4375-4280-8ccb-2957456b68df-kube-api-access-sbbfp\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:22.698380 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.698329    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/660875c4-4375-4280-8ccb-2957456b68df-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:22.700415 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.700397    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660875c4-4375-4280-8ccb-2957456b68df-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:22.709879 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.709854    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbbfp\" (UniqueName: \"kubernetes.io/projected/660875c4-4375-4280-8ccb-2957456b68df-kube-api-access-sbbfp\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:22.793934 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.793916    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:22.926982 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.926923    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"]
Jul 07 02:40:22.929251 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:40:22.929220    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660875c4_4375_4280_8ccb_2957456b68df.slice/crio-1611c6877c448def095ab8ae97b0f52a489d980445213bac0dcc487dfd4bad4e WatchSource:0}: Error finding container 1611c6877c448def095ab8ae97b0f52a489d980445213bac0dcc487dfd4bad4e: Status 404 returned error can't find the container with id 1611c6877c448def095ab8ae97b0f52a489d980445213bac0dcc487dfd4bad4e
Jul 07 02:40:22.988942 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.988919    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft" event={"ID":"660875c4-4375-4280-8ccb-2957456b68df","Type":"ContainerStarted","Data":"8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061"}
Jul 07 02:40:22.989042 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:22.988961    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft" event={"ID":"660875c4-4375-4280-8ccb-2957456b68df","Type":"ContainerStarted","Data":"1611c6877c448def095ab8ae97b0f52a489d980445213bac0dcc487dfd4bad4e"}
Jul 07 02:40:23.994898 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:23.994869    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:40:24.012678 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:24.012626    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft" podStartSLOduration=2.012611238 podStartE2EDuration="2.012611238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:40:24.011472231 +0000 UTC m=+1359.419061919" watchObservedRunningTime="2026-07-07 02:40:24.012611238 +0000 UTC m=+1359.420200939"
Jul 07 02:40:55.001222 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:40:55.001191    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:41:02.251754 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.251717    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"]
Jul 07 02:41:02.256494 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.256478    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.259134 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.259115    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 07 02:41:02.264513 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.264493    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"]
Jul 07 02:41:02.296788 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.296765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.296913 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.296826    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.296913 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.296880    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4k9h\" (UniqueName: \"kubernetes.io/projected/e5411883-34a0-443d-b909-d44adc694128-kube-api-access-l4k9h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.297185 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.296938    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.297185 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.296988    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.297185 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.297011    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5411883-34a0-443d-b909-d44adc694128-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.398053 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.398027    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4k9h\" (UniqueName: \"kubernetes.io/projected/e5411883-34a0-443d-b909-d44adc694128-kube-api-access-l4k9h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.398162 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.398062    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.398162 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.398091    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.398162 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.398112    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5411883-34a0-443d-b909-d44adc694128-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.398162 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.398146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.398401 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.398176    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.398540 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.398514    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.398613 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.398527    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.398613 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.398582    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.400340 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.400319    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.400699 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.400679    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5411883-34a0-443d-b909-d44adc694128-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.407175 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.407157    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4k9h\" (UniqueName: \"kubernetes.io/projected/e5411883-34a0-443d-b909-d44adc694128-kube-api-access-l4k9h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.567222 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.567199    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:02.702269 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:02.702241    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"]
Jul 07 02:41:02.705045 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:41:02.705022    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5411883_34a0_443d_b909_d44adc694128.slice/crio-3de0f7f5e3ef8c0fb6b99c868e68942a6ba2386e783b46ef6939a02f88b3542d WatchSource:0}: Error finding container 3de0f7f5e3ef8c0fb6b99c868e68942a6ba2386e783b46ef6939a02f88b3542d: Status 404 returned error can't find the container with id 3de0f7f5e3ef8c0fb6b99c868e68942a6ba2386e783b46ef6939a02f88b3542d
Jul 07 02:41:03.180704 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:03.180670    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m" event={"ID":"e5411883-34a0-443d-b909-d44adc694128","Type":"ContainerStarted","Data":"cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66"}
Jul 07 02:41:03.180875 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:03.180718    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m" event={"ID":"e5411883-34a0-443d-b909-d44adc694128","Type":"ContainerStarted","Data":"3de0f7f5e3ef8c0fb6b99c868e68942a6ba2386e783b46ef6939a02f88b3542d"}
Jul 07 02:41:03.205103 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:03.205060    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m" podStartSLOduration=1.205046027 podStartE2EDuration="1.205046027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:41:03.203479944 +0000 UTC m=+1398.611069631" watchObservedRunningTime="2026-07-07 02:41:03.205046027 +0000 UTC m=+1398.612635746"
Jul 07 02:41:03.567837 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:03.567811    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:12.567536 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:12.567500    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:12.580180 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:12.580153    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:13.241605 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:13.241580    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:41:29.778784 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:29.778747    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"]
Jul 07 02:41:29.783060 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:29.783039    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:29.785418 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:29.785397    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:41:29.785636 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:29.785411    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-djztz\""
Jul 07 02:41:29.785744 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:29.785484    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 07 02:41:29.794444 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:29.793951    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"]
Jul 07 02:41:29.919961 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:29.919934    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:29.920116 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:29.919991    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cffc7f-a17f-4213-8a92-e4108046fed2-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:29.920116 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:29.920034    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:29.920219 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:29.920112    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdl9w\" (UniqueName: \"kubernetes.io/projected/b5cffc7f-a17f-4213-8a92-e4108046fed2-kube-api-access-cdl9w\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:30.020793 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.020760    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:30.020929 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.020808    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cffc7f-a17f-4213-8a92-e4108046fed2-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:30.020929 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.020845    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:30.020929 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.020909    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdl9w\" (UniqueName: \"kubernetes.io/projected/b5cffc7f-a17f-4213-8a92-e4108046fed2-kube-api-access-cdl9w\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:30.021150 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.021129    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:30.021222 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.021180    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:30.023526 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.023505    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cffc7f-a17f-4213-8a92-e4108046fed2-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:30.029457 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.029408    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdl9w\" (UniqueName: \"kubernetes.io/projected/b5cffc7f-a17f-4213-8a92-e4108046fed2-kube-api-access-cdl9w\") pod \"auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:30.097206 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.097186    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:30.238662 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.238635    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"]
Jul 07 02:41:30.239687 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:41:30.239650    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cffc7f_a17f_4213_8a92_e4108046fed2.slice/crio-e6900969ad5122ae0ae13739f225679b4240c9562d5c361de2f616ffa6407db1 WatchSource:0}: Error finding container e6900969ad5122ae0ae13739f225679b4240c9562d5c361de2f616ffa6407db1: Status 404 returned error can't find the container with id e6900969ad5122ae0ae13739f225679b4240c9562d5c361de2f616ffa6407db1
Jul 07 02:41:30.314669 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.314643    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n" event={"ID":"b5cffc7f-a17f-4213-8a92-e4108046fed2","Type":"ContainerStarted","Data":"96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc"}
Jul 07 02:41:30.314754 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:30.314691    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n" event={"ID":"b5cffc7f-a17f-4213-8a92-e4108046fed2","Type":"ContainerStarted","Data":"e6900969ad5122ae0ae13739f225679b4240c9562d5c361de2f616ffa6407db1"}
Jul 07 02:41:31.320452 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:31.320424    2570 generic.go:356] "Generic (PLEG): container finished" podID="b5cffc7f-a17f-4213-8a92-e4108046fed2" containerID="96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc" exitCode=0
Jul 07 02:41:31.320809 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:31.320482    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n" event={"ID":"b5cffc7f-a17f-4213-8a92-e4108046fed2","Type":"ContainerDied","Data":"96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc"}
Jul 07 02:41:32.327364 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:32.327312    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n" event={"ID":"b5cffc7f-a17f-4213-8a92-e4108046fed2","Type":"ContainerStarted","Data":"fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594"}
Jul 07 02:41:32.327798 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:32.327440    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:41:32.347496 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:32.347449    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n" podStartSLOduration=2.710139272 podStartE2EDuration="3.347434818s" totalImagesPullingTime="0s" totalInitContainerRuntime="637.295546ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:41:32.346477693 +0000 UTC m=+1427.754067381" watchObservedRunningTime="2026-07-07 02:41:32.347434818 +0000 UTC m=+1427.755024504"
Jul 07 02:41:45.061284 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:41:45.060848    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"]
Jul 07 02:42:00.083446 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.083386    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m" podUID="e5411883-34a0-443d-b909-d44adc694128" containerName="main" containerID="cri-o://cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66" gracePeriod=45
Jul 07 02:42:00.231050 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.231007    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m" podUID="e5411883-34a0-443d-b909-d44adc694128" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 07 02:42:00.340753 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.340694    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:42:00.374877 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.374854    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-home\") pod \"e5411883-34a0-443d-b909-d44adc694128\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") "
Jul 07 02:42:00.374987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.374893    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-model-cache\") pod \"e5411883-34a0-443d-b909-d44adc694128\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") "
Jul 07 02:42:00.374987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.374921    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4k9h\" (UniqueName: \"kubernetes.io/projected/e5411883-34a0-443d-b909-d44adc694128-kube-api-access-l4k9h\") pod \"e5411883-34a0-443d-b909-d44adc694128\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") "
Jul 07 02:42:00.374987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.374957    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5411883-34a0-443d-b909-d44adc694128-tls-certs\") pod \"e5411883-34a0-443d-b909-d44adc694128\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") "
Jul 07 02:42:00.374987 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.374981    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-tmp-dir\") pod \"e5411883-34a0-443d-b909-d44adc694128\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") "
Jul 07 02:42:00.375209 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.375029    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-dshm\") pod \"e5411883-34a0-443d-b909-d44adc694128\" (UID: \"e5411883-34a0-443d-b909-d44adc694128\") "
Jul 07 02:42:00.375267 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.375207    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5411883-34a0-443d-b909-d44adc694128" (UID: "e5411883-34a0-443d-b909-d44adc694128"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:42:00.375399 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.375374    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5411883-34a0-443d-b909-d44adc694128" (UID: "e5411883-34a0-443d-b909-d44adc694128"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:42:00.375502 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.375449    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:42:00.375558 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.375500    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-home" (OuterVolumeSpecName: "home") pod "e5411883-34a0-443d-b909-d44adc694128" (UID: "e5411883-34a0-443d-b909-d44adc694128"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:42:00.377306 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.377280    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-dshm" (OuterVolumeSpecName: "dshm") pod "e5411883-34a0-443d-b909-d44adc694128" (UID: "e5411883-34a0-443d-b909-d44adc694128"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:42:00.377306 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.377289    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5411883-34a0-443d-b909-d44adc694128-kube-api-access-l4k9h" (OuterVolumeSpecName: "kube-api-access-l4k9h") pod "e5411883-34a0-443d-b909-d44adc694128" (UID: "e5411883-34a0-443d-b909-d44adc694128"). InnerVolumeSpecName "kube-api-access-l4k9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:42:00.377513 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.377290    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5411883-34a0-443d-b909-d44adc694128-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5411883-34a0-443d-b909-d44adc694128" (UID: "e5411883-34a0-443d-b909-d44adc694128"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:42:00.469549 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.469517    2570 generic.go:356] "Generic (PLEG): container finished" podID="e5411883-34a0-443d-b909-d44adc694128" containerID="cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66" exitCode=0
Jul 07 02:42:00.469641 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.469564    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"
Jul 07 02:42:00.469641 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.469600    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m" event={"ID":"e5411883-34a0-443d-b909-d44adc694128","Type":"ContainerDied","Data":"cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66"}
Jul 07 02:42:00.469724 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.469649    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m" event={"ID":"e5411883-34a0-443d-b909-d44adc694128","Type":"ContainerDied","Data":"3de0f7f5e3ef8c0fb6b99c868e68942a6ba2386e783b46ef6939a02f88b3542d"}
Jul 07 02:42:00.469724 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.469687    2570 scope.go:117] "RemoveContainer" containerID="cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66"
Jul 07 02:42:00.475842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.475820    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5411883-34a0-443d-b909-d44adc694128-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:42:00.475944 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.475868    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-tmp-dir\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:42:00.475944 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.475881    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-dshm\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:42:00.475944 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.475892    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5411883-34a0-443d-b909-d44adc694128-home\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:42:00.475944 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.475904    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4k9h\" (UniqueName: \"kubernetes.io/projected/e5411883-34a0-443d-b909-d44adc694128-kube-api-access-l4k9h\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:42:00.480756 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.480533    2570 scope.go:117] "RemoveContainer" containerID="cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66"
Jul 07 02:42:00.481558 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:42:00.481535    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66\": container with ID starting with cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66 not found: ID does not exist" containerID="cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66"
Jul 07 02:42:00.481643 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.481565    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66"} err="failed to get container status \"cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66\": rpc error: code = NotFound desc = could not find container \"cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66\": container with ID starting with cd2bf501504e7715b0b78808ddb6cc71aac182bd5f0a83e67f2c4cbe9498ba66 not found: ID does not exist"
Jul 07 02:42:00.491541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.491521    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"]
Jul 07 02:42:00.495430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:00.495413    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmlb4m"]
Jul 07 02:42:01.086512 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:01.086482    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5411883-34a0-443d-b909-d44adc694128" path="/var/lib/kubelet/pods/e5411883-34a0-443d-b909-d44adc694128/volumes"
Jul 07 02:42:03.335225 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:03.335198    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:42:05.376331 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.376298    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"]
Jul 07 02:42:05.386331 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:42:05.386300    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:42:05.386331 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:42:05.386331 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:42:05.386331 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061" cmd=["/bin/sleep","15"]
Jul 07 02:42:05.386588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.386361    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft" message="" err=<
Jul 07 02:42:05.386588 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:42:05.386588 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:42:05.386588 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:42:05.386588 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:42:05.386374    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:42:05.386588 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:42:05.386588 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:42:05.386588 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft" podUID="660875c4-4375-4280-8ccb-2957456b68df" containerName="main" containerID="cri-o://8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061"
Jul 07 02:42:05.386588 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.386416    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft" podUID="660875c4-4375-4280-8ccb-2957456b68df" containerName="main" containerID="cri-o://8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061" gracePeriod=60
Jul 07 02:42:05.635327 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.635275    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:42:05.716200 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.716172    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/660875c4-4375-4280-8ccb-2957456b68df-tokenizer-uds\") pod \"660875c4-4375-4280-8ccb-2957456b68df\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") "
Jul 07 02:42:05.716368 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.716238    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660875c4-4375-4280-8ccb-2957456b68df-tls-certs\") pod \"660875c4-4375-4280-8ccb-2957456b68df\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") "
Jul 07 02:42:05.716368 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.716299    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbbfp\" (UniqueName: \"kubernetes.io/projected/660875c4-4375-4280-8ccb-2957456b68df-kube-api-access-sbbfp\") pod \"660875c4-4375-4280-8ccb-2957456b68df\" (UID: \"660875c4-4375-4280-8ccb-2957456b68df\") "
Jul 07 02:42:05.716511 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.716489    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660875c4-4375-4280-8ccb-2957456b68df-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "660875c4-4375-4280-8ccb-2957456b68df" (UID: "660875c4-4375-4280-8ccb-2957456b68df"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:42:05.716626 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.716606    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/660875c4-4375-4280-8ccb-2957456b68df-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:42:05.718413 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.718390    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660875c4-4375-4280-8ccb-2957456b68df-kube-api-access-sbbfp" (OuterVolumeSpecName: "kube-api-access-sbbfp") pod "660875c4-4375-4280-8ccb-2957456b68df" (UID: "660875c4-4375-4280-8ccb-2957456b68df"). InnerVolumeSpecName "kube-api-access-sbbfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:42:05.718522 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.718503    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660875c4-4375-4280-8ccb-2957456b68df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "660875c4-4375-4280-8ccb-2957456b68df" (UID: "660875c4-4375-4280-8ccb-2957456b68df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:42:05.817834 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.817802    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/660875c4-4375-4280-8ccb-2957456b68df-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:42:05.817834 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:05.817833    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbbfp\" (UniqueName: \"kubernetes.io/projected/660875c4-4375-4280-8ccb-2957456b68df-kube-api-access-sbbfp\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:42:06.501468 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:06.501438    2570 generic.go:356] "Generic (PLEG): container finished" podID="660875c4-4375-4280-8ccb-2957456b68df" containerID="8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061" exitCode=0
Jul 07 02:42:06.501865 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:06.501484    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"
Jul 07 02:42:06.501865 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:06.501494    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft" event={"ID":"660875c4-4375-4280-8ccb-2957456b68df","Type":"ContainerDied","Data":"8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061"}
Jul 07 02:42:06.501865 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:06.501533    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft" event={"ID":"660875c4-4375-4280-8ccb-2957456b68df","Type":"ContainerDied","Data":"1611c6877c448def095ab8ae97b0f52a489d980445213bac0dcc487dfd4bad4e"}
Jul 07 02:42:06.501865 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:06.501553    2570 scope.go:117] "RemoveContainer" containerID="8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061"
Jul 07 02:42:06.510662 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:06.510646    2570 scope.go:117] "RemoveContainer" containerID="8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061"
Jul 07 02:42:06.510886 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:42:06.510868    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061\": container with ID starting with 8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061 not found: ID does not exist" containerID="8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061"
Jul 07 02:42:06.510930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:06.510894    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061"} err="failed to get container status \"8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061\": rpc error: code = NotFound desc = could not find container \"8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061\": container with ID starting with 8831deb2762cd50b25ddb9ab2e7a232e547266de5180726916d1009ee17e2061 not found: ID does not exist"
Jul 07 02:42:06.522579 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:06.522560    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"]
Jul 07 02:42:06.527209 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:06.527191    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche6gbft"]
Jul 07 02:42:07.087681 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:07.087648    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="660875c4-4375-4280-8ccb-2957456b68df" path="/var/lib/kubelet/pods/660875c4-4375-4280-8ccb-2957456b68df/volumes"
Jul 07 02:42:33.264529 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.264500    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"]
Jul 07 02:42:33.265002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.264868    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660875c4-4375-4280-8ccb-2957456b68df" containerName="main"
Jul 07 02:42:33.265002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.264879    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="660875c4-4375-4280-8ccb-2957456b68df" containerName="main"
Jul 07 02:42:33.265002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.264889    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5411883-34a0-443d-b909-d44adc694128" containerName="main"
Jul 07 02:42:33.265002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.264895    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5411883-34a0-443d-b909-d44adc694128" containerName="main"
Jul 07 02:42:33.265002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.264954    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5411883-34a0-443d-b909-d44adc694128" containerName="main"
Jul 07 02:42:33.265002 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.264962    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="660875c4-4375-4280-8ccb-2957456b68df" containerName="main"
Jul 07 02:42:33.268026 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.268007    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.270561 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.270541    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-6p4tg\""
Jul 07 02:42:33.270690 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.270577    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 07 02:42:33.279741 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.279719    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"]
Jul 07 02:42:33.340203 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.340183    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.340321 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.340225    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.340401 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.340319    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s5xl\" (UniqueName: \"kubernetes.io/projected/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kube-api-access-5s5xl\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.340448 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.340423    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.441736 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.441712    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s5xl\" (UniqueName: \"kubernetes.io/projected/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kube-api-access-5s5xl\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.441839 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.441760    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.441839 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.441779    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.441839 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.441809    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.442146 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.442123    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.442183 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.442128    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.444465 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.444445    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.450235 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.450213    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s5xl\" (UniqueName: \"kubernetes.io/projected/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kube-api-access-5s5xl\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.577941 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.577919    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:33.717508 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:33.717479    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"]
Jul 07 02:42:33.719590 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:42:33.719566    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23cb441_e8b0_4b5f_83d1_6fa6621463ea.slice/crio-95eb5acb078db0d2cddaf59e6fd51f6f77d9ab6547d3be2a9d180c9708a7bed0 WatchSource:0}: Error finding container 95eb5acb078db0d2cddaf59e6fd51f6f77d9ab6547d3be2a9d180c9708a7bed0: Status 404 returned error can't find the container with id 95eb5acb078db0d2cddaf59e6fd51f6f77d9ab6547d3be2a9d180c9708a7bed0
Jul 07 02:42:34.652684 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:34.652641    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4" event={"ID":"f23cb441-e8b0-4b5f-83d1-6fa6621463ea","Type":"ContainerStarted","Data":"3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f"}
Jul 07 02:42:34.653072 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:34.652735    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4" event={"ID":"f23cb441-e8b0-4b5f-83d1-6fa6621463ea","Type":"ContainerStarted","Data":"95eb5acb078db0d2cddaf59e6fd51f6f77d9ab6547d3be2a9d180c9708a7bed0"}
Jul 07 02:42:35.659413 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:35.659369    2570 generic.go:356] "Generic (PLEG): container finished" podID="f23cb441-e8b0-4b5f-83d1-6fa6621463ea" containerID="3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f" exitCode=0
Jul 07 02:42:35.659413 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:35.659408    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4" event={"ID":"f23cb441-e8b0-4b5f-83d1-6fa6621463ea","Type":"ContainerDied","Data":"3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f"}
Jul 07 02:42:36.665604 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:36.665575    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4" event={"ID":"f23cb441-e8b0-4b5f-83d1-6fa6621463ea","Type":"ContainerStarted","Data":"1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051"}
Jul 07 02:42:36.665992 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:36.665712    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:42:36.687652 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:42:36.687610    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4" podStartSLOduration=2.624924758 podStartE2EDuration="3.687594476s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.062669718s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:42:36.686381825 +0000 UTC m=+1492.093971512" watchObservedRunningTime="2026-07-07 02:42:36.687594476 +0000 UTC m=+1492.095184163"
Jul 07 02:43:07.672628 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:07.672558    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:43:45.353517 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.353481    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"]
Jul 07 02:43:45.362889 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:43:45.362855    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:43:45.362889 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:43:45.362889 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:43:45.362889 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594" cmd=["/bin/sleep","15"]
Jul 07 02:43:45.363155 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.362903    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n" message="" err=<
Jul 07 02:43:45.363155 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:43:45.363155 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:43:45.363155 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:43:45.363155 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:43:45.362915    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:43:45.363155 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:43:45.363155 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:43:45.363155 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n" podUID="b5cffc7f-a17f-4213-8a92-e4108046fed2" containerName="main" containerID="cri-o://fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594"
Jul 07 02:43:45.363155 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.362960    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n" podUID="b5cffc7f-a17f-4213-8a92-e4108046fed2" containerName="main" containerID="cri-o://fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594" gracePeriod=60
Jul 07 02:43:45.617026 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.616970    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:43:45.733572 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.733548    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-kserve-provision-location\") pod \"b5cffc7f-a17f-4213-8a92-e4108046fed2\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") "
Jul 07 02:43:45.733709 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.733623    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cffc7f-a17f-4213-8a92-e4108046fed2-tls-certs\") pod \"b5cffc7f-a17f-4213-8a92-e4108046fed2\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") "
Jul 07 02:43:45.733709 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.733645    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdl9w\" (UniqueName: \"kubernetes.io/projected/b5cffc7f-a17f-4213-8a92-e4108046fed2-kube-api-access-cdl9w\") pod \"b5cffc7f-a17f-4213-8a92-e4108046fed2\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") "
Jul 07 02:43:45.733786 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.733718    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-tokenizer-uds\") pod \"b5cffc7f-a17f-4213-8a92-e4108046fed2\" (UID: \"b5cffc7f-a17f-4213-8a92-e4108046fed2\") "
Jul 07 02:43:45.734004 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.733984    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b5cffc7f-a17f-4213-8a92-e4108046fed2" (UID: "b5cffc7f-a17f-4213-8a92-e4108046fed2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:43:45.734342 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.734282    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5cffc7f-a17f-4213-8a92-e4108046fed2" (UID: "b5cffc7f-a17f-4213-8a92-e4108046fed2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:43:45.735801 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.735782    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5cffc7f-a17f-4213-8a92-e4108046fed2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5cffc7f-a17f-4213-8a92-e4108046fed2" (UID: "b5cffc7f-a17f-4213-8a92-e4108046fed2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:43:45.735898 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.735857    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cffc7f-a17f-4213-8a92-e4108046fed2-kube-api-access-cdl9w" (OuterVolumeSpecName: "kube-api-access-cdl9w") pod "b5cffc7f-a17f-4213-8a92-e4108046fed2" (UID: "b5cffc7f-a17f-4213-8a92-e4108046fed2"). InnerVolumeSpecName "kube-api-access-cdl9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:43:45.834847 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.834821    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:43:45.834847 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.834844    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5cffc7f-a17f-4213-8a92-e4108046fed2-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:43:45.834973 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.834854    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5cffc7f-a17f-4213-8a92-e4108046fed2-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:43:45.834973 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:45.834864    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdl9w\" (UniqueName: \"kubernetes.io/projected/b5cffc7f-a17f-4213-8a92-e4108046fed2-kube-api-access-cdl9w\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:43:46.004888 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.004811    2570 generic.go:356] "Generic (PLEG): container finished" podID="b5cffc7f-a17f-4213-8a92-e4108046fed2" containerID="fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594" exitCode=0
Jul 07 02:43:46.004888 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.004872    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"
Jul 07 02:43:46.005076 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.004899    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n" event={"ID":"b5cffc7f-a17f-4213-8a92-e4108046fed2","Type":"ContainerDied","Data":"fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594"}
Jul 07 02:43:46.005076 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.004957    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n" event={"ID":"b5cffc7f-a17f-4213-8a92-e4108046fed2","Type":"ContainerDied","Data":"e6900969ad5122ae0ae13739f225679b4240c9562d5c361de2f616ffa6407db1"}
Jul 07 02:43:46.005076 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.004979    2570 scope.go:117] "RemoveContainer" containerID="fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594"
Jul 07 02:43:46.014721 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.014706    2570 scope.go:117] "RemoveContainer" containerID="96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc"
Jul 07 02:43:46.022400 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.022383    2570 scope.go:117] "RemoveContainer" containerID="fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594"
Jul 07 02:43:46.022634 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:43:46.022615    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594\": container with ID starting with fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594 not found: ID does not exist" containerID="fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594"
Jul 07 02:43:46.022697 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.022640    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594"} err="failed to get container status \"fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594\": rpc error: code = NotFound desc = could not find container \"fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594\": container with ID starting with fd3592685830b9fb75f79c41646f99d70bde7319a3efaea9846fb8df006a5594 not found: ID does not exist"
Jul 07 02:43:46.022697 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.022657    2570 scope.go:117] "RemoveContainer" containerID="96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc"
Jul 07 02:43:46.022882 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:43:46.022866    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc\": container with ID starting with 96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc not found: ID does not exist" containerID="96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc"
Jul 07 02:43:46.022954 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.022892    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc"} err="failed to get container status \"96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc\": rpc error: code = NotFound desc = could not find container \"96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc\": container with ID starting with 96324f83c651c5468a67f3f4eeb2f9a52f0f4ae00cdf8695d7ad802e512c91dc not found: ID does not exist"
Jul 07 02:43:46.026325 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.026302    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"]
Jul 07 02:43:46.031507 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:46.031486    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-58d5d5d4b6-ghg8n"]
Jul 07 02:43:47.087098 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:43:47.087061    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5cffc7f-a17f-4213-8a92-e4108046fed2" path="/var/lib/kubelet/pods/b5cffc7f-a17f-4213-8a92-e4108046fed2/volumes"
Jul 07 02:44:29.614647 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.614607    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"]
Jul 07 02:44:29.625180 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:44:29.625147    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:44:29.625180 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:44:29.625180 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:44:29.625180 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051" cmd=["/bin/sleep","15"]
Jul 07 02:44:29.625520 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.625195    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4" message="" err=<
Jul 07 02:44:29.625520 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:44:29.625520 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:44:29.625520 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:44:29.625520 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:44:29.625206    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:44:29.625520 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:44:29.625520 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:44:29.625520 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4" podUID="f23cb441-e8b0-4b5f-83d1-6fa6621463ea" containerName="main" containerID="cri-o://1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051"
Jul 07 02:44:29.625520 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.625242    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4" podUID="f23cb441-e8b0-4b5f-83d1-6fa6621463ea" containerName="main" containerID="cri-o://1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051" gracePeriod=60
Jul 07 02:44:29.887994 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.887962    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:44:29.988952 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.988927    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tokenizer-uds\") pod \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") "
Jul 07 02:44:29.989115 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.989037    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kserve-provision-location\") pod \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") "
Jul 07 02:44:29.989115 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.989062    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tls-certs\") pod \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") "
Jul 07 02:44:29.989115 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.989103    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s5xl\" (UniqueName: \"kubernetes.io/projected/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kube-api-access-5s5xl\") pod \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\" (UID: \"f23cb441-e8b0-4b5f-83d1-6fa6621463ea\") "
Jul 07 02:44:29.989230 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.989206    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f23cb441-e8b0-4b5f-83d1-6fa6621463ea" (UID: "f23cb441-e8b0-4b5f-83d1-6fa6621463ea"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:29.989340 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.989327    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:44:29.989820 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.989752    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f23cb441-e8b0-4b5f-83d1-6fa6621463ea" (UID: "f23cb441-e8b0-4b5f-83d1-6fa6621463ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:29.991316 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.991297    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f23cb441-e8b0-4b5f-83d1-6fa6621463ea" (UID: "f23cb441-e8b0-4b5f-83d1-6fa6621463ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:44:29.991431 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:29.991373    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kube-api-access-5s5xl" (OuterVolumeSpecName: "kube-api-access-5s5xl") pod "f23cb441-e8b0-4b5f-83d1-6fa6621463ea" (UID: "f23cb441-e8b0-4b5f-83d1-6fa6621463ea"). InnerVolumeSpecName "kube-api-access-5s5xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:44:30.090644 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.090622    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s5xl\" (UniqueName: \"kubernetes.io/projected/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kube-api-access-5s5xl\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:44:30.090740 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.090647    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:44:30.090740 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.090686    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f23cb441-e8b0-4b5f-83d1-6fa6621463ea-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:44:30.218493 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.218425    2570 generic.go:356] "Generic (PLEG): container finished" podID="f23cb441-e8b0-4b5f-83d1-6fa6621463ea" containerID="1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051" exitCode=0
Jul 07 02:44:30.218493 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.218484    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"
Jul 07 02:44:30.218634 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.218507    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4" event={"ID":"f23cb441-e8b0-4b5f-83d1-6fa6621463ea","Type":"ContainerDied","Data":"1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051"}
Jul 07 02:44:30.218634 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.218559    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4" event={"ID":"f23cb441-e8b0-4b5f-83d1-6fa6621463ea","Type":"ContainerDied","Data":"95eb5acb078db0d2cddaf59e6fd51f6f77d9ab6547d3be2a9d180c9708a7bed0"}
Jul 07 02:44:30.218634 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.218580    2570 scope.go:117] "RemoveContainer" containerID="1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051"
Jul 07 02:44:30.227930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.227915    2570 scope.go:117] "RemoveContainer" containerID="3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f"
Jul 07 02:44:30.235632 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.235617    2570 scope.go:117] "RemoveContainer" containerID="1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051"
Jul 07 02:44:30.235874 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:44:30.235854    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051\": container with ID starting with 1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051 not found: ID does not exist" containerID="1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051"
Jul 07 02:44:30.235912 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.235881    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051"} err="failed to get container status \"1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051\": rpc error: code = NotFound desc = could not find container \"1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051\": container with ID starting with 1f035d05bf4fed4a79dd77241fe29e2cc7f6730e3d6f19684eff6210a25d3051 not found: ID does not exist"
Jul 07 02:44:30.235912 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.235898    2570 scope.go:117] "RemoveContainer" containerID="3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f"
Jul 07 02:44:30.236136 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:44:30.236113    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f\": container with ID starting with 3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f not found: ID does not exist" containerID="3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f"
Jul 07 02:44:30.236233 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.236141    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f"} err="failed to get container status \"3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f\": rpc error: code = NotFound desc = could not find container \"3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f\": container with ID starting with 3e697329766f932d575f3d68bae73d8f62b5d03f0b9ffd0d68c18afa9450764f not found: ID does not exist"
Jul 07 02:44:30.238818 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.238729    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"]
Jul 07 02:44:30.240786 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:30.240769    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqp7r4"]
Jul 07 02:44:31.092282 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:31.092188    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f23cb441-e8b0-4b5f-83d1-6fa6621463ea" path="/var/lib/kubelet/pods/f23cb441-e8b0-4b5f-83d1-6fa6621463ea/volumes"
Jul 07 02:44:54.176489 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.176460    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"]
Jul 07 02:44:54.176892 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.176813    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cffc7f-a17f-4213-8a92-e4108046fed2" containerName="main"
Jul 07 02:44:54.176892 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.176825    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cffc7f-a17f-4213-8a92-e4108046fed2" containerName="main"
Jul 07 02:44:54.176892 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.176833    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23cb441-e8b0-4b5f-83d1-6fa6621463ea" containerName="storage-initializer"
Jul 07 02:44:54.176892 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.176839    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23cb441-e8b0-4b5f-83d1-6fa6621463ea" containerName="storage-initializer"
Jul 07 02:44:54.176892 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.176847    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cffc7f-a17f-4213-8a92-e4108046fed2" containerName="storage-initializer"
Jul 07 02:44:54.176892 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.176853    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cffc7f-a17f-4213-8a92-e4108046fed2" containerName="storage-initializer"
Jul 07 02:44:54.176892 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.176860    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23cb441-e8b0-4b5f-83d1-6fa6621463ea" containerName="main"
Jul 07 02:44:54.176892 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.176866    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23cb441-e8b0-4b5f-83d1-6fa6621463ea" containerName="main"
Jul 07 02:44:54.177140 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.176948    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5cffc7f-a17f-4213-8a92-e4108046fed2" containerName="main"
Jul 07 02:44:54.177140 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.176957    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f23cb441-e8b0-4b5f-83d1-6fa6621463ea" containerName="main"
Jul 07 02:44:54.181118 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.181102    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.183448 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.183423    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:44:54.183706 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.183685    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-8z94m\""
Jul 07 02:44:54.183816 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.183737    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:44:54.184290 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.184274    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:44:54.184406 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.184338    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:44:54.184406 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.184368    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 07 02:44:54.189024 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.189003    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"]
Jul 07 02:44:54.279744 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.279721    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjvh\" (UniqueName: \"kubernetes.io/projected/466cd61d-f798-4921-801e-3b41039303d3-kube-api-access-ppjvh\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.279855 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.279758    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/466cd61d-f798-4921-801e-3b41039303d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.279855 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.279777    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.279965 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.279946    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.381081 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.381055    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjvh\" (UniqueName: \"kubernetes.io/projected/466cd61d-f798-4921-801e-3b41039303d3-kube-api-access-ppjvh\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.381208 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.381097    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/466cd61d-f798-4921-801e-3b41039303d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.381208 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.381114    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.381282 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.381230    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.381467 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.381452    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.381581 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.381563    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.383657 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.383637    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/466cd61d-f798-4921-801e-3b41039303d3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.389862 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.389834    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjvh\" (UniqueName: \"kubernetes.io/projected/466cd61d-f798-4921-801e-3b41039303d3-kube-api-access-ppjvh\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.491672 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.491589    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:54.630100 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:54.629996    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"]
Jul 07 02:44:54.632965 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:44:54.632933    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466cd61d_f798_4921_801e_3b41039303d3.slice/crio-9be43bf18257eaf1d64b4d8f89dbf30ae23306f94f7b029da13257ff438138db WatchSource:0}: Error finding container 9be43bf18257eaf1d64b4d8f89dbf30ae23306f94f7b029da13257ff438138db: Status 404 returned error can't find the container with id 9be43bf18257eaf1d64b4d8f89dbf30ae23306f94f7b029da13257ff438138db
Jul 07 02:44:55.341484 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:55.341447    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v" event={"ID":"466cd61d-f798-4921-801e-3b41039303d3","Type":"ContainerStarted","Data":"b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf"}
Jul 07 02:44:55.341822 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:55.341497    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v" event={"ID":"466cd61d-f798-4921-801e-3b41039303d3","Type":"ContainerStarted","Data":"9be43bf18257eaf1d64b4d8f89dbf30ae23306f94f7b029da13257ff438138db"}
Jul 07 02:44:56.348457 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:56.348422    2570 generic.go:356] "Generic (PLEG): container finished" podID="466cd61d-f798-4921-801e-3b41039303d3" containerID="b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf" exitCode=0
Jul 07 02:44:56.348844 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:56.348508    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v" event={"ID":"466cd61d-f798-4921-801e-3b41039303d3","Type":"ContainerDied","Data":"b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf"}
Jul 07 02:44:56.348844 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:56.348555    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v" event={"ID":"466cd61d-f798-4921-801e-3b41039303d3","Type":"ContainerStarted","Data":"77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9"}
Jul 07 02:44:56.348844 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:56.348589    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:44:56.367874 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:44:56.367834    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v" podStartSLOduration=1.731939386 podStartE2EDuration="2.367817803s" totalImagesPullingTime="0s" totalInitContainerRuntime="635.878417ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:44:56.366335102 +0000 UTC m=+1631.773924789" watchObservedRunningTime="2026-07-07 02:44:56.367817803 +0000 UTC m=+1631.775407514"
Jul 07 02:45:27.355770 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:45:27.355739    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:46:15.958541 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:15.958505    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"]
Jul 07 02:46:15.968577 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:46:15.968544    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:46:15.968577 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:46:15.968577 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:46:15.968577 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9" cmd=["/bin/sleep","15"]
Jul 07 02:46:15.968900 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:15.968590    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v" message="" err=<
Jul 07 02:46:15.968900 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:46:15.968900 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:46:15.968900 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:46:15.968900 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:46:15.968605    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:46:15.968900 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:46:15.968900 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:46:15.968900 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v" podUID="466cd61d-f798-4921-801e-3b41039303d3" containerName="main" containerID="cri-o://77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9"
Jul 07 02:46:15.968900 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:15.968643    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v" podUID="466cd61d-f798-4921-801e-3b41039303d3" containerName="main" containerID="cri-o://77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9" gracePeriod=60
Jul 07 02:46:16.214431 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.214375    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:46:16.330643 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.330616    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppjvh\" (UniqueName: \"kubernetes.io/projected/466cd61d-f798-4921-801e-3b41039303d3-kube-api-access-ppjvh\") pod \"466cd61d-f798-4921-801e-3b41039303d3\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") "
Jul 07 02:46:16.330774 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.330672    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/466cd61d-f798-4921-801e-3b41039303d3-tls-certs\") pod \"466cd61d-f798-4921-801e-3b41039303d3\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") "
Jul 07 02:46:16.330774 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.330702    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-kserve-provision-location\") pod \"466cd61d-f798-4921-801e-3b41039303d3\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") "
Jul 07 02:46:16.330774 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.330731    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-tokenizer-uds\") pod \"466cd61d-f798-4921-801e-3b41039303d3\" (UID: \"466cd61d-f798-4921-801e-3b41039303d3\") "
Jul 07 02:46:16.331048 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.331027    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "466cd61d-f798-4921-801e-3b41039303d3" (UID: "466cd61d-f798-4921-801e-3b41039303d3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:46:16.331464 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.331402    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "466cd61d-f798-4921-801e-3b41039303d3" (UID: "466cd61d-f798-4921-801e-3b41039303d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:46:16.332894 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.332865    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466cd61d-f798-4921-801e-3b41039303d3-kube-api-access-ppjvh" (OuterVolumeSpecName: "kube-api-access-ppjvh") pod "466cd61d-f798-4921-801e-3b41039303d3" (UID: "466cd61d-f798-4921-801e-3b41039303d3"). InnerVolumeSpecName "kube-api-access-ppjvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:46:16.332980 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.332924    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466cd61d-f798-4921-801e-3b41039303d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "466cd61d-f798-4921-801e-3b41039303d3" (UID: "466cd61d-f798-4921-801e-3b41039303d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:46:16.431774 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.431750    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppjvh\" (UniqueName: \"kubernetes.io/projected/466cd61d-f798-4921-801e-3b41039303d3-kube-api-access-ppjvh\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:46:16.431774 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.431775    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/466cd61d-f798-4921-801e-3b41039303d3-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:46:16.431774 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.431786    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:46:16.431973 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.431794    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/466cd61d-f798-4921-801e-3b41039303d3-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:46:16.747510 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.747477    2570 generic.go:356] "Generic (PLEG): container finished" podID="466cd61d-f798-4921-801e-3b41039303d3" containerID="77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9" exitCode=0
Jul 07 02:46:16.747686 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.747527    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"
Jul 07 02:46:16.747686 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.747561    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v" event={"ID":"466cd61d-f798-4921-801e-3b41039303d3","Type":"ContainerDied","Data":"77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9"}
Jul 07 02:46:16.747686 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.747617    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v" event={"ID":"466cd61d-f798-4921-801e-3b41039303d3","Type":"ContainerDied","Data":"9be43bf18257eaf1d64b4d8f89dbf30ae23306f94f7b029da13257ff438138db"}
Jul 07 02:46:16.747686 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.747641    2570 scope.go:117] "RemoveContainer" containerID="77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9"
Jul 07 02:46:16.757270 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.757250    2570 scope.go:117] "RemoveContainer" containerID="b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf"
Jul 07 02:46:16.765068 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.765054    2570 scope.go:117] "RemoveContainer" containerID="77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9"
Jul 07 02:46:16.765311 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:46:16.765290    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9\": container with ID starting with 77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9 not found: ID does not exist" containerID="77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9"
Jul 07 02:46:16.765412 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.765322    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9"} err="failed to get container status \"77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9\": rpc error: code = NotFound desc = could not find container \"77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9\": container with ID starting with 77aaedb2cbf891292b7a49efc0875bb0062662d51cc516a7565d8abff6b8e7f9 not found: ID does not exist"
Jul 07 02:46:16.765412 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.765361    2570 scope.go:117] "RemoveContainer" containerID="b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf"
Jul 07 02:46:16.765629 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:46:16.765607    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf\": container with ID starting with b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf not found: ID does not exist" containerID="b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf"
Jul 07 02:46:16.765685 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.765638    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf"} err="failed to get container status \"b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf\": rpc error: code = NotFound desc = could not find container \"b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf\": container with ID starting with b34044ed1113c433c39c865d7faa5478660a4a70044e29f9891d95abbcccbebf not found: ID does not exist"
Jul 07 02:46:16.769266 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.769237    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"]
Jul 07 02:46:16.772919 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:16.772896    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecfc4v"]
Jul 07 02:46:17.086583 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:17.086560    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="466cd61d-f798-4921-801e-3b41039303d3" path="/var/lib/kubelet/pods/466cd61d-f798-4921-801e-3b41039303d3/volumes"
Jul 07 02:46:39.489377 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.489330    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"]
Jul 07 02:46:39.489746 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.489697    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="466cd61d-f798-4921-801e-3b41039303d3" containerName="main"
Jul 07 02:46:39.489746 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.489708    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="466cd61d-f798-4921-801e-3b41039303d3" containerName="main"
Jul 07 02:46:39.489820 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.489748    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="466cd61d-f798-4921-801e-3b41039303d3" containerName="storage-initializer"
Jul 07 02:46:39.489820 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.489755    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="466cd61d-f798-4921-801e-3b41039303d3" containerName="storage-initializer"
Jul 07 02:46:39.489820 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.489814    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="466cd61d-f798-4921-801e-3b41039303d3" containerName="main"
Jul 07 02:46:39.494159 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.494143    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:39.497844 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.497823    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:46:39.497985 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.497865    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:46:39.498655 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.498638    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:46:39.502651 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.502632    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"]
Jul 07 02:46:39.617896 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.617870    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km7f5\" (UniqueName: \"kubernetes.io/projected/b14d1ff1-3a46-4631-b1cc-9a73657d5408-kube-api-access-km7f5\") pod \"e2e-pvc-model-download-cbftb\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:39.718984 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.718956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km7f5\" (UniqueName: \"kubernetes.io/projected/b14d1ff1-3a46-4631-b1cc-9a73657d5408-kube-api-access-km7f5\") pod \"e2e-pvc-model-download-cbftb\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:39.727525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:39.727499    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km7f5\" (UniqueName: \"kubernetes.io/projected/b14d1ff1-3a46-4631-b1cc-9a73657d5408-kube-api-access-km7f5\") pod \"e2e-pvc-model-download-cbftb\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:42.039399 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.039336    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"e2e-pvc-model-download-cbftb\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:42.044184 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.044163    2570 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"e2e-pvc-model-download-cbftb\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:42.140154 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.140119    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"e2e-pvc-model-download-cbftb\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:42.140290 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.140246    2570 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"e2e-pvc-model-download-cbftb\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:42.143145 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.143128    2570 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"e2e-pvc-model-download-cbftb\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") DevicePath \"csi-b3de9d333c49122025fa55d45536993dab31593b80601716898d0124996442e3\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:42.356993 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.356970    2570 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"e2e-pvc-model-download-cbftb\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4d4e79e722613b74efd0a3ee1474df31ab6a7717dd4a75fe1631c9eba9ff0a8b/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:42.370905 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.370884    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"e2e-pvc-model-download-cbftb\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:42.502905 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.502885    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:42.633168 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.633144    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"]
Jul 07 02:46:42.635386 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:46:42.635337    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb14d1ff1_3a46_4631_b1cc_9a73657d5408.slice/crio-24738415bc6a9cf33e60784bce5327ae75812d196dd401fa5aae04971c7b4a8d WatchSource:0}: Error finding container 24738415bc6a9cf33e60784bce5327ae75812d196dd401fa5aae04971c7b4a8d: Status 404 returned error can't find the container with id 24738415bc6a9cf33e60784bce5327ae75812d196dd401fa5aae04971c7b4a8d
Jul 07 02:46:42.878182 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.878145    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb" event={"ID":"b14d1ff1-3a46-4631-b1cc-9a73657d5408","Type":"ContainerStarted","Data":"ca85675f87c385ff5528da941f50d972115656376cc5dbb08f5fe3805c4679a6"}
Jul 07 02:46:42.878182 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.878190    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb" event={"ID":"b14d1ff1-3a46-4631-b1cc-9a73657d5408","Type":"ContainerStarted","Data":"24738415bc6a9cf33e60784bce5327ae75812d196dd401fa5aae04971c7b4a8d"}
Jul 07 02:46:42.893514 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:42.893418    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb" podStartSLOduration=6.893402046 podStartE2EDuration="6.893402046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:46:42.892698903 +0000 UTC m=+1738.300288612" watchObservedRunningTime="2026-07-07 02:46:42.893402046 +0000 UTC m=+1738.300991732"
Jul 07 02:46:49.916548 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:49.916514    2570 generic.go:356] "Generic (PLEG): container finished" podID="b14d1ff1-3a46-4631-b1cc-9a73657d5408" containerID="ca85675f87c385ff5528da941f50d972115656376cc5dbb08f5fe3805c4679a6" exitCode=0
Jul 07 02:46:49.917029 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:49.916589    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb" event={"ID":"b14d1ff1-3a46-4631-b1cc-9a73657d5408","Type":"ContainerDied","Data":"ca85675f87c385ff5528da941f50d972115656376cc5dbb08f5fe3805c4679a6"}
Jul 07 02:46:51.071038 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.071016    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:51.219253 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.219179    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km7f5\" (UniqueName: \"kubernetes.io/projected/b14d1ff1-3a46-4631-b1cc-9a73657d5408-kube-api-access-km7f5\") pod \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") "
Jul 07 02:46:51.219420 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.219305    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\" (UID: \"b14d1ff1-3a46-4631-b1cc-9a73657d5408\") "
Jul 07 02:46:51.221430 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.221399    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14d1ff1-3a46-4631-b1cc-9a73657d5408-kube-api-access-km7f5" (OuterVolumeSpecName: "kube-api-access-km7f5") pod "b14d1ff1-3a46-4631-b1cc-9a73657d5408" (UID: "b14d1ff1-3a46-4631-b1cc-9a73657d5408"). InnerVolumeSpecName "kube-api-access-km7f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:46:51.223326 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.223307    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8" (OuterVolumeSpecName: "model-storage") pod "b14d1ff1-3a46-4631-b1cc-9a73657d5408" (UID: "b14d1ff1-3a46-4631-b1cc-9a73657d5408"). InnerVolumeSpecName "pvc-72dabf56-560c-4abb-87d3-214aad97d3e0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 07 02:46:51.320812 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.320780    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km7f5\" (UniqueName: \"kubernetes.io/projected/b14d1ff1-3a46-4631-b1cc-9a73657d5408-kube-api-access-km7f5\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:46:51.320946 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.320830    2570 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") on node \"ip-10-0-141-229.ec2.internal\" "
Jul 07 02:46:51.427833 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.427815    2570 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-72dabf56-560c-4abb-87d3-214aad97d3e0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8") on node "ip-10-0-141-229.ec2.internal"
Jul 07 02:46:51.522281 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.522223    2570 reconciler_common.go:299] "Volume detached for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"csi-b3de9d333c49122025fa55d45536993dab31593b80601716898d0124996442e3\""
Jul 07 02:46:51.927166 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.927139    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb"
Jul 07 02:46:51.927305 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.927160    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-cbftb" event={"ID":"b14d1ff1-3a46-4631-b1cc-9a73657d5408","Type":"ContainerDied","Data":"24738415bc6a9cf33e60784bce5327ae75812d196dd401fa5aae04971c7b4a8d"}
Jul 07 02:46:51.927305 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:46:51.927196    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24738415bc6a9cf33e60784bce5327ae75812d196dd401fa5aae04971c7b4a8d"
Jul 07 02:47:02.462948 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.462910    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"]
Jul 07 02:47:02.463565 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.463543    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b14d1ff1-3a46-4631-b1cc-9a73657d5408" containerName="storage-initializer"
Jul 07 02:47:02.463565 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.463565    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b14d1ff1-3a46-4631-b1cc-9a73657d5408" containerName="storage-initializer"
Jul 07 02:47:02.463709 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.463671    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b14d1ff1-3a46-4631-b1cc-9a73657d5408" containerName="storage-initializer"
Jul 07 02:47:02.467561 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.467544    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:02.470189 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.470167    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-tmpcn\""
Jul 07 02:47:02.470325 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.470206    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:47:02.470703 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.470684    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:47:02.470792 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.470742    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:47:02.470847 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.470815    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 07 02:47:02.477790 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.477767    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"]
Jul 07 02:47:02.507907 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.507881    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8kc\" (UniqueName: \"kubernetes.io/projected/4319d23e-f516-440c-909c-289b86a461d2-kube-api-access-4g8kc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:02.508015 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.507911    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4319d23e-f516-440c-909c-289b86a461d2-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:02.508015 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.507930    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4319d23e-f516-440c-909c-289b86a461d2-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:02.608842 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.608821    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8kc\" (UniqueName: \"kubernetes.io/projected/4319d23e-f516-440c-909c-289b86a461d2-kube-api-access-4g8kc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:02.608949 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.608850    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4319d23e-f516-440c-909c-289b86a461d2-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:02.608949 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.608868    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4319d23e-f516-440c-909c-289b86a461d2-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:02.609171 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.609156    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4319d23e-f516-440c-909c-289b86a461d2-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:02.611560 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.611538    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4319d23e-f516-440c-909c-289b86a461d2-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:02.617071 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.617042    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8kc\" (UniqueName: \"kubernetes.io/projected/4319d23e-f516-440c-909c-289b86a461d2-kube-api-access-4g8kc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:02.779154 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.779134    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:02.911034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.911011    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"]
Jul 07 02:47:02.913453 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:47:02.913425    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4319d23e_f516_440c_909c_289b86a461d2.slice/crio-cd7e8e1127cbdb5c998643c6ca207bb1be97a2686c3a0d28a38e66c2ae05fd34 WatchSource:0}: Error finding container cd7e8e1127cbdb5c998643c6ca207bb1be97a2686c3a0d28a38e66c2ae05fd34: Status 404 returned error can't find the container with id cd7e8e1127cbdb5c998643c6ca207bb1be97a2686c3a0d28a38e66c2ae05fd34
Jul 07 02:47:02.980644 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.980606    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j" event={"ID":"4319d23e-f516-440c-909c-289b86a461d2","Type":"ContainerStarted","Data":"77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508"}
Jul 07 02:47:02.980748 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.980657    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j" event={"ID":"4319d23e-f516-440c-909c-289b86a461d2","Type":"ContainerStarted","Data":"cd7e8e1127cbdb5c998643c6ca207bb1be97a2686c3a0d28a38e66c2ae05fd34"}
Jul 07 02:47:02.980805 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:02.980748    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:47:03.001191 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:03.001147    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j" podStartSLOduration=1.001133237 podStartE2EDuration="1.001133237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:47:03.00012073 +0000 UTC m=+1758.407710437" watchObservedRunningTime="2026-07-07 02:47:03.001133237 +0000 UTC m=+1758.408722924"
Jul 07 02:47:33.986729 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:47:33.986698    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:49:09.599452 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:09.599364    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"]
Jul 07 02:49:09.608704 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:49:09.608672    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:49:09.608704 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:49:09.608704 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:49:09.608704 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508" cmd=["/bin/sleep","15"]
Jul 07 02:49:09.608918 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:09.608721    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j" message="" err=<
Jul 07 02:49:09.608918 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:49:09.608918 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:49:09.608918 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:49:09.608918 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:49:09.608734    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:49:09.608918 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:49:09.608918 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:49:09.608918 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j" podUID="4319d23e-f516-440c-909c-289b86a461d2" containerName="main" containerID="cri-o://77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508"
Jul 07 02:49:09.608918 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:09.608779    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j" podUID="4319d23e-f516-440c-909c-289b86a461d2" containerName="main" containerID="cri-o://77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508" gracePeriod=60
Jul 07 02:49:09.858688 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:09.858635    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:49:09.994548 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:09.994521    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4319d23e-f516-440c-909c-289b86a461d2-tokenizer-uds\") pod \"4319d23e-f516-440c-909c-289b86a461d2\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") "
Jul 07 02:49:09.994691 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:09.994585    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g8kc\" (UniqueName: \"kubernetes.io/projected/4319d23e-f516-440c-909c-289b86a461d2-kube-api-access-4g8kc\") pod \"4319d23e-f516-440c-909c-289b86a461d2\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") "
Jul 07 02:49:09.994691 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:09.994628    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4319d23e-f516-440c-909c-289b86a461d2-tls-certs\") pod \"4319d23e-f516-440c-909c-289b86a461d2\" (UID: \"4319d23e-f516-440c-909c-289b86a461d2\") "
Jul 07 02:49:09.994824 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:09.994776    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4319d23e-f516-440c-909c-289b86a461d2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4319d23e-f516-440c-909c-289b86a461d2" (UID: "4319d23e-f516-440c-909c-289b86a461d2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:49:09.996737 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:09.996707    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4319d23e-f516-440c-909c-289b86a461d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4319d23e-f516-440c-909c-289b86a461d2" (UID: "4319d23e-f516-440c-909c-289b86a461d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:49:09.996737 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:09.996721    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4319d23e-f516-440c-909c-289b86a461d2-kube-api-access-4g8kc" (OuterVolumeSpecName: "kube-api-access-4g8kc") pod "4319d23e-f516-440c-909c-289b86a461d2" (UID: "4319d23e-f516-440c-909c-289b86a461d2"). InnerVolumeSpecName "kube-api-access-4g8kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:49:10.096081 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.096059    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4319d23e-f516-440c-909c-289b86a461d2-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:49:10.096081 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.096080    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4319d23e-f516-440c-909c-289b86a461d2-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:49:10.096214 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.096089    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8kc\" (UniqueName: \"kubernetes.io/projected/4319d23e-f516-440c-909c-289b86a461d2-kube-api-access-4g8kc\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:49:10.580338 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.580306    2570 generic.go:356] "Generic (PLEG): container finished" podID="4319d23e-f516-440c-909c-289b86a461d2" containerID="77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508" exitCode=0
Jul 07 02:49:10.580525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.580390    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"
Jul 07 02:49:10.580525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.580403    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j" event={"ID":"4319d23e-f516-440c-909c-289b86a461d2","Type":"ContainerDied","Data":"77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508"}
Jul 07 02:49:10.580525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.580470    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j" event={"ID":"4319d23e-f516-440c-909c-289b86a461d2","Type":"ContainerDied","Data":"cd7e8e1127cbdb5c998643c6ca207bb1be97a2686c3a0d28a38e66c2ae05fd34"}
Jul 07 02:49:10.580525 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.580496    2570 scope.go:117] "RemoveContainer" containerID="77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508"
Jul 07 02:49:10.590283 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.590266    2570 scope.go:117] "RemoveContainer" containerID="77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508"
Jul 07 02:49:10.590560 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:49:10.590542    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508\": container with ID starting with 77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508 not found: ID does not exist" containerID="77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508"
Jul 07 02:49:10.590618 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.590571    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508"} err="failed to get container status \"77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508\": rpc error: code = NotFound desc = could not find container \"77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508\": container with ID starting with 77c10cc5d799ab861ef88b59b84d966c93bddb7b254ee4b9addbfac1de86d508 not found: ID does not exist"
Jul 07 02:49:10.601867 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.601846    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"]
Jul 07 02:49:10.607727 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:10.607708    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schefbf7j"]
Jul 07 02:49:11.087562 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:11.087528    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4319d23e-f516-440c-909c-289b86a461d2" path="/var/lib/kubelet/pods/4319d23e-f516-440c-909c-289b86a461d2/volumes"
Jul 07 02:49:33.774574 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.774539    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"]
Jul 07 02:49:33.775627 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.775602    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4319d23e-f516-440c-909c-289b86a461d2" containerName="main"
Jul 07 02:49:33.775627 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.775629    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4319d23e-f516-440c-909c-289b86a461d2" containerName="main"
Jul 07 02:49:33.775767 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.775712    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4319d23e-f516-440c-909c-289b86a461d2" containerName="main"
Jul 07 02:49:33.778911 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.778894    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:33.782427 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.782402    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:49:33.782538 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.782427    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 07 02:49:33.782538 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.782476    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:49:33.782538 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.782478    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:49:33.782702 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.782555    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-vz2mt\""
Jul 07 02:49:33.788932 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.788911    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"]
Jul 07 02:49:33.880432 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.880409    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s98p\" (UniqueName: \"kubernetes.io/projected/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-kube-api-access-7s98p\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:33.880528 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.880474    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:33.880528 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.880496    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:33.981227 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.981202    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:33.981322 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.981235    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:33.981322 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.981272    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s98p\" (UniqueName: \"kubernetes.io/projected/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-kube-api-access-7s98p\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:33.981587 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.981570    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:33.983795 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.983777    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:33.988750 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:33.988728    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s98p\" (UniqueName: \"kubernetes.io/projected/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-kube-api-access-7s98p\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:34.088995 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:34.088974    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:34.220448 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:34.220421    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"]
Jul 07 02:49:34.701032 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:34.700999    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw" event={"ID":"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4","Type":"ContainerStarted","Data":"348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181"}
Jul 07 02:49:34.701222 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:34.701046    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw" event={"ID":"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4","Type":"ContainerStarted","Data":"9087e19c3a94619299bc1701086b0602622e52aba3ed65e15057342c68c1fb96"}
Jul 07 02:49:34.701222 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:34.701144    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:49:34.719876 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:49:34.719830    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw" podStartSLOduration=1.719814572 podStartE2EDuration="1.719814572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:49:34.718264806 +0000 UTC m=+1910.125854492" watchObservedRunningTime="2026-07-07 02:49:34.719814572 +0000 UTC m=+1910.127404258"
Jul 07 02:50:05.707481 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:50:05.707446    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:52:27.659041 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:27.658999    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"]
Jul 07 02:52:27.668490 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:52:27.668452    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:52:27.668490 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:52:27.668490 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:52:27.668490 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181" cmd=["/bin/sleep","15"]
Jul 07 02:52:27.668802 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:27.668507    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw" message="" err=<
Jul 07 02:52:27.668802 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:52:27.668802 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:52:27.668802 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:52:27.668802 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:52:27.668518    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:52:27.668802 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:52:27.668802 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:52:27.668802 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw" podUID="1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4" containerName="main" containerID="cri-o://348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181"
Jul 07 02:52:27.668802 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:27.668573    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw" podUID="1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4" containerName="main" containerID="cri-o://348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181" gracePeriod=60
Jul 07 02:52:27.913661 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:27.913606    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:52:28.051342 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.051318    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s98p\" (UniqueName: \"kubernetes.io/projected/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-kube-api-access-7s98p\") pod \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") "
Jul 07 02:52:28.051496 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.051375    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tokenizer-uds\") pod \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") "
Jul 07 02:52:28.051496 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.051422    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tls-certs\") pod \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\" (UID: \"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4\") "
Jul 07 02:52:28.051611 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.051587    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4" (UID: "1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:52:28.053464 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.053435    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-kube-api-access-7s98p" (OuterVolumeSpecName: "kube-api-access-7s98p") pod "1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4" (UID: "1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4"). InnerVolumeSpecName "kube-api-access-7s98p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:52:28.053570 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.053539    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4" (UID: "1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:52:28.152514 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.152483    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s98p\" (UniqueName: \"kubernetes.io/projected/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-kube-api-access-7s98p\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:52:28.152514 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.152513    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:52:28.152514 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.152524    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:52:28.545566 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.545534    2570 generic.go:356] "Generic (PLEG): container finished" podID="1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4" containerID="348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181" exitCode=0
Jul 07 02:52:28.545752 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.545576    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"
Jul 07 02:52:28.545752 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.545635    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw" event={"ID":"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4","Type":"ContainerDied","Data":"348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181"}
Jul 07 02:52:28.545752 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.545682    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw" event={"ID":"1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4","Type":"ContainerDied","Data":"9087e19c3a94619299bc1701086b0602622e52aba3ed65e15057342c68c1fb96"}
Jul 07 02:52:28.545752 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.545702    2570 scope.go:117] "RemoveContainer" containerID="348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181"
Jul 07 02:52:28.557308 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.557288    2570 scope.go:117] "RemoveContainer" containerID="348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181"
Jul 07 02:52:28.557877 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:52:28.557854    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181\": container with ID starting with 348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181 not found: ID does not exist" containerID="348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181"
Jul 07 02:52:28.557978 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.557883    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181"} err="failed to get container status \"348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181\": rpc error: code = NotFound desc = could not find container \"348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181\": container with ID starting with 348e103dbfa8e25852532ba4dfc2ae79fa437f9e49f7fce6109a154f41b24181 not found: ID does not exist"
Jul 07 02:52:28.567789 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.567759    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"]
Jul 07 02:52:28.571403 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:28.571382    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-sche8glrw"]
Jul 07 02:52:29.087785 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:29.087752    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4" path="/var/lib/kubelet/pods/1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4/volumes"
Jul 07 02:52:55.833690 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.833658    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"]
Jul 07 02:52:55.834240 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.834205    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4" containerName="main"
Jul 07 02:52:55.834240 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.834223    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4" containerName="main"
Jul 07 02:52:55.834606 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.834341    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8363bc-73bb-4bbf-8b41-e9c457ab5bf4" containerName="main"
Jul 07 02:52:55.837447 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.837428    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:55.848210 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.848190    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:52:55.848443 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.848423    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:52:55.848499 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.848450    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-d45hz\""
Jul 07 02:52:55.848499 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.848473    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 07 02:52:55.852501 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.852486    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:52:55.857064 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.857043    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"]
Jul 07 02:52:55.874780 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.874758    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af95bc9-0979-478a-8624-1bb068a7d17a-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:55.874866 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.874818    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4af95bc9-0979-478a-8624-1bb068a7d17a-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:55.874866 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.874856    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cftv\" (UniqueName: \"kubernetes.io/projected/4af95bc9-0979-478a-8624-1bb068a7d17a-kube-api-access-6cftv\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:55.975514 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.975494    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af95bc9-0979-478a-8624-1bb068a7d17a-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:55.975613 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.975544    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4af95bc9-0979-478a-8624-1bb068a7d17a-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:55.975613 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.975566    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cftv\" (UniqueName: \"kubernetes.io/projected/4af95bc9-0979-478a-8624-1bb068a7d17a-kube-api-access-6cftv\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:55.975922 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.975900    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4af95bc9-0979-478a-8624-1bb068a7d17a-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:55.978200 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.978181    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af95bc9-0979-478a-8624-1bb068a7d17a-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:55.989621 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:55.989588    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cftv\" (UniqueName: \"kubernetes.io/projected/4af95bc9-0979-478a-8624-1bb068a7d17a-kube-api-access-6cftv\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:56.146023 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:56.145964    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:56.298320 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:56.298296    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"]
Jul 07 02:52:56.300954 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:52:56.300928    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af95bc9_0979_478a_8624_1bb068a7d17a.slice/crio-49c7ec0afce2e8f768dc0e5a93c94d73353a8a60385fd72f5212a50191bebd38 WatchSource:0}: Error finding container 49c7ec0afce2e8f768dc0e5a93c94d73353a8a60385fd72f5212a50191bebd38: Status 404 returned error can't find the container with id 49c7ec0afce2e8f768dc0e5a93c94d73353a8a60385fd72f5212a50191bebd38
Jul 07 02:52:56.684393 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:56.684342    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9" event={"ID":"4af95bc9-0979-478a-8624-1bb068a7d17a","Type":"ContainerStarted","Data":"551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8"}
Jul 07 02:52:56.684555 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:56.684420    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9" event={"ID":"4af95bc9-0979-478a-8624-1bb068a7d17a","Type":"ContainerStarted","Data":"49c7ec0afce2e8f768dc0e5a93c94d73353a8a60385fd72f5212a50191bebd38"}
Jul 07 02:52:56.684555 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:56.684524    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:52:56.739722 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:52:56.739672    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9" podStartSLOduration=1.7396545479999999 podStartE2EDuration="1.739654548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:52:56.737410245 +0000 UTC m=+2112.144999934" watchObservedRunningTime="2026-07-07 02:52:56.739654548 +0000 UTC m=+2112.147244239"
Jul 07 02:53:27.691835 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:53:27.691806    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:56:01.012231 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.012200    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"]
Jul 07 02:56:01.015802 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.015787    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.025793 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.025770    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:56:01.026045 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.026026    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-8zppf\""
Jul 07 02:56:01.026835 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.026819    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 07 02:56:01.034106 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.034083    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"]
Jul 07 02:56:01.147921 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.147900    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.148034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.147947    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4498\" (UniqueName: \"kubernetes.io/projected/13bf1709-f656-4a98-aa37-082c9c375e40-kube-api-access-b4498\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.148034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.147975    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bf1709-f656-4a98-aa37-082c9c375e40-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.148034 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.148020    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.249030 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.248995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bf1709-f656-4a98-aa37-082c9c375e40-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.249130 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.249038    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.249130 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.249108    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.249205 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.249174    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4498\" (UniqueName: \"kubernetes.io/projected/13bf1709-f656-4a98-aa37-082c9c375e40-kube-api-access-b4498\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.249578 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.249546    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.249578 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.249561    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.251606 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.251585    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bf1709-f656-4a98-aa37-082c9c375e40-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.261129 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.261106    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4498\" (UniqueName: \"kubernetes.io/projected/13bf1709-f656-4a98-aa37-082c9c375e40-kube-api-access-b4498\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.325488 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.325462    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:01.459314 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.459290    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"]
Jul 07 02:56:01.461234 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:56:01.461206    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bf1709_f656_4a98_aa37_082c9c375e40.slice/crio-93c0fb48c3df3256d2d72481b74c3d993e82b34d3b3c890daa9d05abcb8568e1 WatchSource:0}: Error finding container 93c0fb48c3df3256d2d72481b74c3d993e82b34d3b3c890daa9d05abcb8568e1: Status 404 returned error can't find the container with id 93c0fb48c3df3256d2d72481b74c3d993e82b34d3b3c890daa9d05abcb8568e1
Jul 07 02:56:01.579821 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.579740    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k" event={"ID":"13bf1709-f656-4a98-aa37-082c9c375e40","Type":"ContainerStarted","Data":"fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28"}
Jul 07 02:56:01.579821 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:01.579795    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k" event={"ID":"13bf1709-f656-4a98-aa37-082c9c375e40","Type":"ContainerStarted","Data":"93c0fb48c3df3256d2d72481b74c3d993e82b34d3b3c890daa9d05abcb8568e1"}
Jul 07 02:56:02.586309 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:02.586272    2570 generic.go:356] "Generic (PLEG): container finished" podID="13bf1709-f656-4a98-aa37-082c9c375e40" containerID="fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28" exitCode=0
Jul 07 02:56:02.586761 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:02.586330    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k" event={"ID":"13bf1709-f656-4a98-aa37-082c9c375e40","Type":"ContainerDied","Data":"fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28"}
Jul 07 02:56:03.592651 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:03.592619    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k" event={"ID":"13bf1709-f656-4a98-aa37-082c9c375e40","Type":"ContainerStarted","Data":"eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24"}
Jul 07 02:56:03.593013 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:03.592757    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:03.613206 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:03.613158    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k" podStartSLOduration=2.979205305 podStartE2EDuration="3.613142098s" totalImagesPullingTime="0s" totalInitContainerRuntime="633.936793ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:56:03.612189567 +0000 UTC m=+2299.019779255" watchObservedRunningTime="2026-07-07 02:56:03.613142098 +0000 UTC m=+2299.020731783"
Jul 07 02:56:18.163033 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.162999    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"]
Jul 07 02:56:18.172822 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:56:18.172784    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:56:18.172822 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:56:18.172822 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:56:18.172822 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8" cmd=["/bin/sleep","15"]
Jul 07 02:56:18.173128 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.172841    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9" message="" err=<
Jul 07 02:56:18.173128 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:56:18.173128 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:56:18.173128 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:56:18.173128 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:56:18.172857    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:56:18.173128 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:56:18.173128 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:56:18.173128 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9" podUID="4af95bc9-0979-478a-8624-1bb068a7d17a" containerName="main" containerID="cri-o://551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8"
Jul 07 02:56:18.173128 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.172915    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9" podUID="4af95bc9-0979-478a-8624-1bb068a7d17a" containerName="main" containerID="cri-o://551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8" gracePeriod=60
Jul 07 02:56:18.438891 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.438865    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:56:18.491221 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.491198    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4af95bc9-0979-478a-8624-1bb068a7d17a-tokenizer-uds\") pod \"4af95bc9-0979-478a-8624-1bb068a7d17a\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") "
Jul 07 02:56:18.491320 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.491235    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cftv\" (UniqueName: \"kubernetes.io/projected/4af95bc9-0979-478a-8624-1bb068a7d17a-kube-api-access-6cftv\") pod \"4af95bc9-0979-478a-8624-1bb068a7d17a\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") "
Jul 07 02:56:18.491320 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.491282    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af95bc9-0979-478a-8624-1bb068a7d17a-tls-certs\") pod \"4af95bc9-0979-478a-8624-1bb068a7d17a\" (UID: \"4af95bc9-0979-478a-8624-1bb068a7d17a\") "
Jul 07 02:56:18.491505 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.491484    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af95bc9-0979-478a-8624-1bb068a7d17a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4af95bc9-0979-478a-8624-1bb068a7d17a" (UID: "4af95bc9-0979-478a-8624-1bb068a7d17a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:56:18.491642 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.491626    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4af95bc9-0979-478a-8624-1bb068a7d17a-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:56:18.493500 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.493477    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af95bc9-0979-478a-8624-1bb068a7d17a-kube-api-access-6cftv" (OuterVolumeSpecName: "kube-api-access-6cftv") pod "4af95bc9-0979-478a-8624-1bb068a7d17a" (UID: "4af95bc9-0979-478a-8624-1bb068a7d17a"). InnerVolumeSpecName "kube-api-access-6cftv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:56:18.493555 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.493488    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af95bc9-0979-478a-8624-1bb068a7d17a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4af95bc9-0979-478a-8624-1bb068a7d17a" (UID: "4af95bc9-0979-478a-8624-1bb068a7d17a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:56:18.592028 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.592003    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4af95bc9-0979-478a-8624-1bb068a7d17a-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:56:18.592028 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.592025    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cftv\" (UniqueName: \"kubernetes.io/projected/4af95bc9-0979-478a-8624-1bb068a7d17a-kube-api-access-6cftv\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:56:18.663437 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.663414    2570 generic.go:356] "Generic (PLEG): container finished" podID="4af95bc9-0979-478a-8624-1bb068a7d17a" containerID="551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8" exitCode=0
Jul 07 02:56:18.663520 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.663444    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9" event={"ID":"4af95bc9-0979-478a-8624-1bb068a7d17a","Type":"ContainerDied","Data":"551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8"}
Jul 07 02:56:18.663520 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.663494    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9" event={"ID":"4af95bc9-0979-478a-8624-1bb068a7d17a","Type":"ContainerDied","Data":"49c7ec0afce2e8f768dc0e5a93c94d73353a8a60385fd72f5212a50191bebd38"}
Jul 07 02:56:18.663520 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.663514    2570 scope.go:117] "RemoveContainer" containerID="551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8"
Jul 07 02:56:18.663624 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.663458    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"
Jul 07 02:56:18.673498 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.673481    2570 scope.go:117] "RemoveContainer" containerID="551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8"
Jul 07 02:56:18.673744 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:56:18.673727    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8\": container with ID starting with 551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8 not found: ID does not exist" containerID="551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8"
Jul 07 02:56:18.673794 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.673751    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8"} err="failed to get container status \"551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8\": rpc error: code = NotFound desc = could not find container \"551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8\": container with ID starting with 551049e496de905a3ce0ab52cba7318f0c7baccff3c09c260a564d29fefba0a8 not found: ID does not exist"
Jul 07 02:56:18.685639 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.685617    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"]
Jul 07 02:56:18.690551 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:18.690530    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-sche6dkc9"]
Jul 07 02:56:19.086988 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:19.086957    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4af95bc9-0979-478a-8624-1bb068a7d17a" path="/var/lib/kubelet/pods/4af95bc9-0979-478a-8624-1bb068a7d17a/volumes"
Jul 07 02:56:34.599464 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:34.599437    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:56:59.660817 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.660782    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"]
Jul 07 02:56:59.661256 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.661148    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af95bc9-0979-478a-8624-1bb068a7d17a" containerName="main"
Jul 07 02:56:59.661256 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.661160    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af95bc9-0979-478a-8624-1bb068a7d17a" containerName="main"
Jul 07 02:56:59.661256 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.661227    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af95bc9-0979-478a-8624-1bb068a7d17a" containerName="main"
Jul 07 02:56:59.669971 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.669935    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.672841 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.672816    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"]
Jul 07 02:56:59.674016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.673985    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-qbvfl\""
Jul 07 02:56:59.674016 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.674000    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 07 02:56:59.829442 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.829417    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fea3a8a-092e-4992-9a17-d3df87664b71-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.829568 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.829452    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96tj\" (UniqueName: \"kubernetes.io/projected/8fea3a8a-092e-4992-9a17-d3df87664b71-kube-api-access-n96tj\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.829568 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.829485    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.829685 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.829557    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.930630 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.930577    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.930630 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.930614    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.930748 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.930667    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fea3a8a-092e-4992-9a17-d3df87664b71-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.930748 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.930689    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96tj\" (UniqueName: \"kubernetes.io/projected/8fea3a8a-092e-4992-9a17-d3df87664b71-kube-api-access-n96tj\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.930965 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.930942    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.931003 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.930970    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.933162 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.933138    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fea3a8a-092e-4992-9a17-d3df87664b71-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.938524 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.938502    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96tj\" (UniqueName: \"kubernetes.io/projected/8fea3a8a-092e-4992-9a17-d3df87664b71-kube-api-access-n96tj\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:56:59.980392 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:56:59.980374    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:57:00.119878 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:57:00.119834    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"]
Jul 07 02:57:00.122361 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:57:00.122313    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fea3a8a_092e_4992_9a17_d3df87664b71.slice/crio-aa1e0f66c583bd917e7c5ba8276b2f9c6401de292ae25fc7e73e29970959bd0a WatchSource:0}: Error finding container aa1e0f66c583bd917e7c5ba8276b2f9c6401de292ae25fc7e73e29970959bd0a: Status 404 returned error can't find the container with id aa1e0f66c583bd917e7c5ba8276b2f9c6401de292ae25fc7e73e29970959bd0a
Jul 07 02:57:00.872071 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:57:00.872045    2570 generic.go:356] "Generic (PLEG): container finished" podID="8fea3a8a-092e-4992-9a17-d3df87664b71" containerID="32068a0c10c4d8ca27a48ac1597c7770b42beaad9ecde11088ee6994e14cca96" exitCode=0
Jul 07 02:57:00.872688 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:57:00.872125    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7" event={"ID":"8fea3a8a-092e-4992-9a17-d3df87664b71","Type":"ContainerDied","Data":"32068a0c10c4d8ca27a48ac1597c7770b42beaad9ecde11088ee6994e14cca96"}
Jul 07 02:57:00.872688 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:57:00.872162    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7" event={"ID":"8fea3a8a-092e-4992-9a17-d3df87664b71","Type":"ContainerStarted","Data":"aa1e0f66c583bd917e7c5ba8276b2f9c6401de292ae25fc7e73e29970959bd0a"}
Jul 07 02:57:01.879209 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:57:01.879172    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7" event={"ID":"8fea3a8a-092e-4992-9a17-d3df87664b71","Type":"ContainerStarted","Data":"4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600"}
Jul 07 02:57:01.879619 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:57:01.879261    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:57:01.900121 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:57:01.900073    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7" podStartSLOduration=2.266673376 podStartE2EDuration="2.900059637s" totalImagesPullingTime="0s" totalInitContainerRuntime="633.386261ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:57:01.898286468 +0000 UTC m=+2357.305876156" watchObservedRunningTime="2026-07-07 02:57:01.900059637 +0000 UTC m=+2357.307649323"
Jul 07 02:57:32.886198 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:57:32.886171    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:58:05.665862 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:05.665826    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"]
Jul 07 02:58:05.675410 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:58:05.675374    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:58:05.675410 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:58:05.675410 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:58:05.675410 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24" cmd=["/bin/sleep","15"]
Jul 07 02:58:05.675714 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:05.675433    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k" message="" err=<
Jul 07 02:58:05.675714 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:58:05.675714 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:58:05.675714 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:58:05.675714 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:58:05.675451    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:58:05.675714 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:58:05.675714 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:58:05.675714 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k" podUID="13bf1709-f656-4a98-aa37-082c9c375e40" containerName="main" containerID="cri-o://eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24"
Jul 07 02:58:05.675714 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:05.675514    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k" podUID="13bf1709-f656-4a98-aa37-082c9c375e40" containerName="main" containerID="cri-o://eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24" gracePeriod=60
Jul 07 02:58:05.921041 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:05.920989    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:58:06.073126 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.073098    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bf1709-f656-4a98-aa37-082c9c375e40-tls-certs\") pod \"13bf1709-f656-4a98-aa37-082c9c375e40\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") "
Jul 07 02:58:06.073269 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.073137    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-tokenizer-uds\") pod \"13bf1709-f656-4a98-aa37-082c9c375e40\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") "
Jul 07 02:58:06.073269 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.073157    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-kserve-provision-location\") pod \"13bf1709-f656-4a98-aa37-082c9c375e40\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") "
Jul 07 02:58:06.073269 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.073211    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4498\" (UniqueName: \"kubernetes.io/projected/13bf1709-f656-4a98-aa37-082c9c375e40-kube-api-access-b4498\") pod \"13bf1709-f656-4a98-aa37-082c9c375e40\" (UID: \"13bf1709-f656-4a98-aa37-082c9c375e40\") "
Jul 07 02:58:06.073483 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.073338    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "13bf1709-f656-4a98-aa37-082c9c375e40" (UID: "13bf1709-f656-4a98-aa37-082c9c375e40"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:58:06.073934 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.073908    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13bf1709-f656-4a98-aa37-082c9c375e40" (UID: "13bf1709-f656-4a98-aa37-082c9c375e40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:58:06.075467 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.075444    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bf1709-f656-4a98-aa37-082c9c375e40-kube-api-access-b4498" (OuterVolumeSpecName: "kube-api-access-b4498") pod "13bf1709-f656-4a98-aa37-082c9c375e40" (UID: "13bf1709-f656-4a98-aa37-082c9c375e40"). InnerVolumeSpecName "kube-api-access-b4498". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:58:06.075522 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.075474    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13bf1709-f656-4a98-aa37-082c9c375e40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13bf1709-f656-4a98-aa37-082c9c375e40" (UID: "13bf1709-f656-4a98-aa37-082c9c375e40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:58:06.174122 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.174060    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4498\" (UniqueName: \"kubernetes.io/projected/13bf1709-f656-4a98-aa37-082c9c375e40-kube-api-access-b4498\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:58:06.174122 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.174086    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bf1709-f656-4a98-aa37-082c9c375e40-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:58:06.174122 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.174099    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:58:06.174122 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.174111    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13bf1709-f656-4a98-aa37-082c9c375e40-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:58:06.194289 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.194259    2570 generic.go:356] "Generic (PLEG): container finished" podID="13bf1709-f656-4a98-aa37-082c9c375e40" containerID="eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24" exitCode=0
Jul 07 02:58:06.194408 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.194310    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"
Jul 07 02:58:06.194408 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.194311    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k" event={"ID":"13bf1709-f656-4a98-aa37-082c9c375e40","Type":"ContainerDied","Data":"eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24"}
Jul 07 02:58:06.194408 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.194387    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k" event={"ID":"13bf1709-f656-4a98-aa37-082c9c375e40","Type":"ContainerDied","Data":"93c0fb48c3df3256d2d72481b74c3d993e82b34d3b3c890daa9d05abcb8568e1"}
Jul 07 02:58:06.194535 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.194416    2570 scope.go:117] "RemoveContainer" containerID="eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24"
Jul 07 02:58:06.203432 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.203415    2570 scope.go:117] "RemoveContainer" containerID="fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28"
Jul 07 02:58:06.211158 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.211143    2570 scope.go:117] "RemoveContainer" containerID="eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24"
Jul 07 02:58:06.211440 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:58:06.211420    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24\": container with ID starting with eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24 not found: ID does not exist" containerID="eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24"
Jul 07 02:58:06.211507 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.211447    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24"} err="failed to get container status \"eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24\": rpc error: code = NotFound desc = could not find container \"eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24\": container with ID starting with eeebcf3f6cd08b270bd2350fd81d28003c30bd4128cde8e6d68ddaa4afe43f24 not found: ID does not exist"
Jul 07 02:58:06.211507 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.211463    2570 scope.go:117] "RemoveContainer" containerID="fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28"
Jul 07 02:58:06.211692 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:58:06.211673    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28\": container with ID starting with fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28 not found: ID does not exist" containerID="fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28"
Jul 07 02:58:06.211736 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.211698    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28"} err="failed to get container status \"fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28\": rpc error: code = NotFound desc = could not find container \"fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28\": container with ID starting with fdbd1a3d5b734635c1db488cce018ed454c1a9a2ee82f9ccdb61b0a961272d28 not found: ID does not exist"
Jul 07 02:58:06.215596 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.215576    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"]
Jul 07 02:58:06.221970 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:06.221952    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schejtb4k"]
Jul 07 02:58:07.086640 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:07.086598    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13bf1709-f656-4a98-aa37-082c9c375e40" path="/var/lib/kubelet/pods/13bf1709-f656-4a98-aa37-082c9c375e40/volumes"
Jul 07 02:58:32.354245 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.354209    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"]
Jul 07 02:58:32.354759 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.354623    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13bf1709-f656-4a98-aa37-082c9c375e40" containerName="main"
Jul 07 02:58:32.354759 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.354636    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="13bf1709-f656-4a98-aa37-082c9c375e40" containerName="main"
Jul 07 02:58:32.354759 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.354661    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13bf1709-f656-4a98-aa37-082c9c375e40" containerName="storage-initializer"
Jul 07 02:58:32.354759 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.354670    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="13bf1709-f656-4a98-aa37-082c9c375e40" containerName="storage-initializer"
Jul 07 02:58:32.354759 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.354740    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="13bf1709-f656-4a98-aa37-082c9c375e40" containerName="main"
Jul 07 02:58:32.357923 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.357906    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.360470 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.360449    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-shrgz\""
Jul 07 02:58:32.361111 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.361096    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 07 02:58:32.369012 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.368992    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"]
Jul 07 02:58:32.480327 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.480302    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.480468 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.480335    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.480468 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.480374    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657kw\" (UniqueName: \"kubernetes.io/projected/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kube-api-access-657kw\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.480590 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.480534    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.581293 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.581272    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.581414 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.581297    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657kw\" (UniqueName: \"kubernetes.io/projected/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kube-api-access-657kw\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.581414 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.581362    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.581414 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.581400    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.581674 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.581658    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.581743 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.581723    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.583921 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.583901    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.589738 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.589723    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657kw\" (UniqueName: \"kubernetes.io/projected/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kube-api-access-657kw\") pod \"custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.666591 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.666535    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:32.804988 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:32.804962    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"]
Jul 07 02:58:32.806095 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:58:32.806070    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f97d80_c6eb_40f4_a1ea_4feb7e1ffb92.slice/crio-95444df6a5a91cff6f0f1fe4344cbe3690f7e448de6e0a67a18483db7c9c2157 WatchSource:0}: Error finding container 95444df6a5a91cff6f0f1fe4344cbe3690f7e448de6e0a67a18483db7c9c2157: Status 404 returned error can't find the container with id 95444df6a5a91cff6f0f1fe4344cbe3690f7e448de6e0a67a18483db7c9c2157
Jul 07 02:58:33.325367 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:33.325303    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc" event={"ID":"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92","Type":"ContainerStarted","Data":"537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b"}
Jul 07 02:58:33.325565 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:33.325377    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc" event={"ID":"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92","Type":"ContainerStarted","Data":"95444df6a5a91cff6f0f1fe4344cbe3690f7e448de6e0a67a18483db7c9c2157"}
Jul 07 02:58:34.332819 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:34.332739    2570 generic.go:356] "Generic (PLEG): container finished" podID="30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" containerID="537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b" exitCode=0
Jul 07 02:58:34.333176 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:34.332824    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc" event={"ID":"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92","Type":"ContainerDied","Data":"537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b"}
Jul 07 02:58:35.339080 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:35.339039    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc" event={"ID":"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92","Type":"ContainerStarted","Data":"3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a"}
Jul 07 02:58:35.339479 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:35.339203    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:58:35.357674 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:58:35.357635    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc" podStartSLOduration=2.731558731 podStartE2EDuration="3.357621969s" totalImagesPullingTime="0s" totalInitContainerRuntime="626.063238ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:58:35.356115152 +0000 UTC m=+2450.763704839" watchObservedRunningTime="2026-07-07 02:58:35.357621969 +0000 UTC m=+2450.765211655"
Jul 07 02:59:00.807720 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:00.807684    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"]
Jul 07 02:59:00.817068 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:59:00.817034    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:59:00.817068 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:59:00.817068 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:59:00.817068 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600" cmd=["/bin/sleep","15"]
Jul 07 02:59:00.817407 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:00.817082    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7" message="" err=<
Jul 07 02:59:00.817407 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:59:00.817407 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:59:00.817407 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 02:59:00.817407 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:59:00.817095    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:59:00.817407 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:59:00.817407 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 02:59:00.817407 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7" podUID="8fea3a8a-092e-4992-9a17-d3df87664b71" containerName="main" containerID="cri-o://4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600"
Jul 07 02:59:00.817407 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:00.817128    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7" podUID="8fea3a8a-092e-4992-9a17-d3df87664b71" containerName="main" containerID="cri-o://4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600" gracePeriod=60
Jul 07 02:59:01.067773 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.067714    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:59:01.218517 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.218486    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-kserve-provision-location\") pod \"8fea3a8a-092e-4992-9a17-d3df87664b71\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") "
Jul 07 02:59:01.218671 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.218553    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fea3a8a-092e-4992-9a17-d3df87664b71-tls-certs\") pod \"8fea3a8a-092e-4992-9a17-d3df87664b71\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") "
Jul 07 02:59:01.218671 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.218585    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n96tj\" (UniqueName: \"kubernetes.io/projected/8fea3a8a-092e-4992-9a17-d3df87664b71-kube-api-access-n96tj\") pod \"8fea3a8a-092e-4992-9a17-d3df87664b71\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") "
Jul 07 02:59:01.218671 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.218600    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-tokenizer-uds\") pod \"8fea3a8a-092e-4992-9a17-d3df87664b71\" (UID: \"8fea3a8a-092e-4992-9a17-d3df87664b71\") "
Jul 07 02:59:01.218959 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.218931    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8fea3a8a-092e-4992-9a17-d3df87664b71" (UID: "8fea3a8a-092e-4992-9a17-d3df87664b71"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:59:01.219284 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.219257    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fea3a8a-092e-4992-9a17-d3df87664b71" (UID: "8fea3a8a-092e-4992-9a17-d3df87664b71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:59:01.220905 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.220884    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fea3a8a-092e-4992-9a17-d3df87664b71-kube-api-access-n96tj" (OuterVolumeSpecName: "kube-api-access-n96tj") pod "8fea3a8a-092e-4992-9a17-d3df87664b71" (UID: "8fea3a8a-092e-4992-9a17-d3df87664b71"). InnerVolumeSpecName "kube-api-access-n96tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:59:01.220986 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.220959    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fea3a8a-092e-4992-9a17-d3df87664b71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fea3a8a-092e-4992-9a17-d3df87664b71" (UID: "8fea3a8a-092e-4992-9a17-d3df87664b71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:59:01.319506 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.319446    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fea3a8a-092e-4992-9a17-d3df87664b71-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:59:01.319506 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.319473    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n96tj\" (UniqueName: \"kubernetes.io/projected/8fea3a8a-092e-4992-9a17-d3df87664b71-kube-api-access-n96tj\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:59:01.319506 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.319488    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:59:01.319506 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.319497    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fea3a8a-092e-4992-9a17-d3df87664b71-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 02:59:01.464116 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.464084    2570 generic.go:356] "Generic (PLEG): container finished" podID="8fea3a8a-092e-4992-9a17-d3df87664b71" containerID="4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600" exitCode=0
Jul 07 02:59:01.464237 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.464134    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"
Jul 07 02:59:01.464237 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.464163    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7" event={"ID":"8fea3a8a-092e-4992-9a17-d3df87664b71","Type":"ContainerDied","Data":"4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600"}
Jul 07 02:59:01.464237 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.464214    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7" event={"ID":"8fea3a8a-092e-4992-9a17-d3df87664b71","Type":"ContainerDied","Data":"aa1e0f66c583bd917e7c5ba8276b2f9c6401de292ae25fc7e73e29970959bd0a"}
Jul 07 02:59:01.464237 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.464235    2570 scope.go:117] "RemoveContainer" containerID="4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600"
Jul 07 02:59:01.474145 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.474128    2570 scope.go:117] "RemoveContainer" containerID="32068a0c10c4d8ca27a48ac1597c7770b42beaad9ecde11088ee6994e14cca96"
Jul 07 02:59:01.482559 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.482541    2570 scope.go:117] "RemoveContainer" containerID="4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600"
Jul 07 02:59:01.482798 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:59:01.482781    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600\": container with ID starting with 4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600 not found: ID does not exist" containerID="4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600"
Jul 07 02:59:01.482852 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.482806    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600"} err="failed to get container status \"4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600\": rpc error: code = NotFound desc = could not find container \"4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600\": container with ID starting with 4a067db88533ba41f02389c33faeb0abc9b7a3cc5d96d81bae6f0741449a9600 not found: ID does not exist"
Jul 07 02:59:01.482852 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.482822    2570 scope.go:117] "RemoveContainer" containerID="32068a0c10c4d8ca27a48ac1597c7770b42beaad9ecde11088ee6994e14cca96"
Jul 07 02:59:01.483074 ip-10-0-141-229 kubenswrapper[2570]: E0707 02:59:01.483056    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32068a0c10c4d8ca27a48ac1597c7770b42beaad9ecde11088ee6994e14cca96\": container with ID starting with 32068a0c10c4d8ca27a48ac1597c7770b42beaad9ecde11088ee6994e14cca96 not found: ID does not exist" containerID="32068a0c10c4d8ca27a48ac1597c7770b42beaad9ecde11088ee6994e14cca96"
Jul 07 02:59:01.483119 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.483079    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32068a0c10c4d8ca27a48ac1597c7770b42beaad9ecde11088ee6994e14cca96"} err="failed to get container status \"32068a0c10c4d8ca27a48ac1597c7770b42beaad9ecde11088ee6994e14cca96\": rpc error: code = NotFound desc = could not find container \"32068a0c10c4d8ca27a48ac1597c7770b42beaad9ecde11088ee6994e14cca96\": container with ID starting with 32068a0c10c4d8ca27a48ac1597c7770b42beaad9ecde11088ee6994e14cca96 not found: ID does not exist"
Jul 07 02:59:01.486217 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.486200    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"]
Jul 07 02:59:01.490490 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:01.490470    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-889t7"]
Jul 07 02:59:03.086402 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:03.086368    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fea3a8a-092e-4992-9a17-d3df87664b71" path="/var/lib/kubelet/pods/8fea3a8a-092e-4992-9a17-d3df87664b71/volumes"
Jul 07 02:59:06.345651 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:06.345624    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 02:59:18.782039 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.782007    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"]
Jul 07 02:59:18.782432 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.782375    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fea3a8a-092e-4992-9a17-d3df87664b71" containerName="storage-initializer"
Jul 07 02:59:18.782432 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.782387    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fea3a8a-092e-4992-9a17-d3df87664b71" containerName="storage-initializer"
Jul 07 02:59:18.782432 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.782410    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fea3a8a-092e-4992-9a17-d3df87664b71" containerName="main"
Jul 07 02:59:18.782432 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.782414    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fea3a8a-092e-4992-9a17-d3df87664b71" containerName="main"
Jul 07 02:59:18.782575 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.782490    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fea3a8a-092e-4992-9a17-d3df87664b71" containerName="main"
Jul 07 02:59:18.787528 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.787508    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.789978 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.789960    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 07 02:59:18.790062 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.789959    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-mv8k6\""
Jul 07 02:59:18.797060 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.797036    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"]
Jul 07 02:59:18.850189 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.850162    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.850371 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.850202    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.850371 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.850222    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.850371 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.850245    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwj4g\" (UniqueName: \"kubernetes.io/projected/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kube-api-access-rwj4g\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.950930 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.950899    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.951060 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.950934    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.951060 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.950969    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwj4g\" (UniqueName: \"kubernetes.io/projected/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kube-api-access-rwj4g\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.951132 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.951078    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.951294 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.951274    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.951385 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.951334    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.953672 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.953654    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:18.959252 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:18.959231    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwj4g\" (UniqueName: \"kubernetes.io/projected/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kube-api-access-rwj4g\") pod \"stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:19.098404 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:19.098322    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:19.231420 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:19.231397    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"]
Jul 07 02:59:19.233606 ip-10-0-141-229 kubenswrapper[2570]: W0707 02:59:19.233572    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa1e220_6c9c_402b_8517_f3301e4ed00c.slice/crio-b69c97f1c7ec3262be13470f5ac5789c6f11a35bdb30a2d3b3f6b16e457d1e24 WatchSource:0}: Error finding container b69c97f1c7ec3262be13470f5ac5789c6f11a35bdb30a2d3b3f6b16e457d1e24: Status 404 returned error can't find the container with id b69c97f1c7ec3262be13470f5ac5789c6f11a35bdb30a2d3b3f6b16e457d1e24
Jul 07 02:59:19.553691 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:19.553660    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf" event={"ID":"8fa1e220-6c9c-402b-8517-f3301e4ed00c","Type":"ContainerStarted","Data":"e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a"}
Jul 07 02:59:19.553875 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:19.553713    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf" event={"ID":"8fa1e220-6c9c-402b-8517-f3301e4ed00c","Type":"ContainerStarted","Data":"b69c97f1c7ec3262be13470f5ac5789c6f11a35bdb30a2d3b3f6b16e457d1e24"}
Jul 07 02:59:20.559798 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:20.559769    2570 generic.go:356] "Generic (PLEG): container finished" podID="8fa1e220-6c9c-402b-8517-f3301e4ed00c" containerID="e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a" exitCode=0
Jul 07 02:59:20.560171 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:20.559837    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf" event={"ID":"8fa1e220-6c9c-402b-8517-f3301e4ed00c","Type":"ContainerDied","Data":"e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a"}
Jul 07 02:59:21.566945 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:21.566906    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf" event={"ID":"8fa1e220-6c9c-402b-8517-f3301e4ed00c","Type":"ContainerStarted","Data":"288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff"}
Jul 07 02:59:21.567439 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:21.567034    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 02:59:21.586415 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:21.586372    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf" podStartSLOduration=2.942692142 podStartE2EDuration="3.586340069s" totalImagesPullingTime="0s" totalInitContainerRuntime="643.647927ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:59:21.584745258 +0000 UTC m=+2496.992334947" watchObservedRunningTime="2026-07-07 02:59:21.586340069 +0000 UTC m=+2496.993929757"
Jul 07 02:59:52.573992 ip-10-0-141-229 kubenswrapper[2570]: I0707 02:59:52.573955    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 03:00:39.338596 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.338558    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"]
Jul 07 03:00:39.350469 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:00:39.350432    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:00:39.350469 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:00:39.350469 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:00:39.350469 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a" cmd=["/bin/sleep","15"]
Jul 07 03:00:39.350783 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.350481    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc" message="" err=<
Jul 07 03:00:39.350783 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:00:39.350783 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:00:39.350783 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:00:39.350783 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:00:39.350494    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:00:39.350783 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:00:39.350783 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:00:39.350783 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc" podUID="30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" containerName="main" containerID="cri-o://3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a"
Jul 07 03:00:39.350783 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.350547    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc" podUID="30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" containerName="main" containerID="cri-o://3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a" gracePeriod=60
Jul 07 03:00:39.606500 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.606478    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 03:00:39.710773 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.710739    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tokenizer-uds\") pod \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") "
Jul 07 03:00:39.710941 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.710794    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kserve-provision-location\") pod \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") "
Jul 07 03:00:39.710941 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.710850    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tls-certs\") pod \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") "
Jul 07 03:00:39.710941 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.710901    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-657kw\" (UniqueName: \"kubernetes.io/projected/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kube-api-access-657kw\") pod \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\" (UID: \"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92\") "
Jul 07 03:00:39.711119 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.710970    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" (UID: "30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:00:39.711180 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.711156    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:00:39.711564 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.711484    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" (UID: "30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:00:39.713192 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.713168    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" (UID: "30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:00:39.713192 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.713173    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kube-api-access-657kw" (OuterVolumeSpecName: "kube-api-access-657kw") pod "30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" (UID: "30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92"). InnerVolumeSpecName "kube-api-access-657kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:00:39.812503 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.812478    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:00:39.812503 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.812501    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:00:39.812628 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.812510    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-657kw\" (UniqueName: \"kubernetes.io/projected/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92-kube-api-access-657kw\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:00:39.952402 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.952319    2570 generic.go:356] "Generic (PLEG): container finished" podID="30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" containerID="3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a" exitCode=0
Jul 07 03:00:39.952402 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.952383    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc" event={"ID":"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92","Type":"ContainerDied","Data":"3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a"}
Jul 07 03:00:39.952529 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.952400    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"
Jul 07 03:00:39.952529 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.952432    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc" event={"ID":"30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92","Type":"ContainerDied","Data":"95444df6a5a91cff6f0f1fe4344cbe3690f7e448de6e0a67a18483db7c9c2157"}
Jul 07 03:00:39.952529 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.952453    2570 scope.go:117] "RemoveContainer" containerID="3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a"
Jul 07 03:00:39.962294 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.962277    2570 scope.go:117] "RemoveContainer" containerID="537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b"
Jul 07 03:00:39.970179 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.970106    2570 scope.go:117] "RemoveContainer" containerID="3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a"
Jul 07 03:00:39.970395 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:00:39.970376    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a\": container with ID starting with 3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a not found: ID does not exist" containerID="3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a"
Jul 07 03:00:39.970452 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.970404    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a"} err="failed to get container status \"3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a\": rpc error: code = NotFound desc = could not find container \"3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a\": container with ID starting with 3b66cff7d2a0de4f892d8c3f6718d291b7c080902663d7f59c2c1ee6a8f06e8a not found: ID does not exist"
Jul 07 03:00:39.970452 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.970421    2570 scope.go:117] "RemoveContainer" containerID="537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b"
Jul 07 03:00:39.970669 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:00:39.970649    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b\": container with ID starting with 537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b not found: ID does not exist" containerID="537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b"
Jul 07 03:00:39.970724 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.970680    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b"} err="failed to get container status \"537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b\": rpc error: code = NotFound desc = could not find container \"537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b\": container with ID starting with 537c792467381d3bcd9e851584733c539ab9b57d4d6223b891710e8b1a49de9b not found: ID does not exist"
Jul 07 03:00:39.975304 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.975283    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"]
Jul 07 03:00:39.981027 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:39.981009    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-7c89bfcfqz6fc"]
Jul 07 03:00:41.087457 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:00:41.087424    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" path="/var/lib/kubelet/pods/30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92/volumes"
Jul 07 03:01:09.321318 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.321243    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"]
Jul 07 03:01:09.321783 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.321683    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" containerName="storage-initializer"
Jul 07 03:01:09.321783 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.321696    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" containerName="storage-initializer"
Jul 07 03:01:09.321783 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.321711    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" containerName="main"
Jul 07 03:01:09.321783 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.321717    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" containerName="main"
Jul 07 03:01:09.321783 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.321776    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="30f97d80-c6eb-40f4-a1ea-4feb7e1ffb92" containerName="main"
Jul 07 03:01:09.324751 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.324734    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.330081 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.330062    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-xhwkr\""
Jul 07 03:01:09.330470 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.330452    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 07 03:01:09.337053 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.337011    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.337164 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.337110    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.337164 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.337127    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"]
Jul 07 03:01:09.337288 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.337174    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndbb4\" (UniqueName: \"kubernetes.io/projected/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kube-api-access-ndbb4\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.337288 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.337210    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.437979 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.437945    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.438144 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.437992    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndbb4\" (UniqueName: \"kubernetes.io/projected/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kube-api-access-ndbb4\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.438144 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.438020    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.438144 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.438072    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.438460 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.438440    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.438460 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.438457    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.440596 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.440573    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.447242 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.447208    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndbb4\" (UniqueName: \"kubernetes.io/projected/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kube-api-access-ndbb4\") pod \"router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.634135 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.634062    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:09.768216 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:09.768193    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"]
Jul 07 03:01:09.771081 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:01:09.771056    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cedec9_11a2_4fc9_9b1f_7ba2c322f86d.slice/crio-6f1fa95bd6e12880bfb282fd1c5ec1036fcc603e64582d19bb57c8d48b357472 WatchSource:0}: Error finding container 6f1fa95bd6e12880bfb282fd1c5ec1036fcc603e64582d19bb57c8d48b357472: Status 404 returned error can't find the container with id 6f1fa95bd6e12880bfb282fd1c5ec1036fcc603e64582d19bb57c8d48b357472
Jul 07 03:01:10.096197 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:10.096162    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2" event={"ID":"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d","Type":"ContainerStarted","Data":"c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11"}
Jul 07 03:01:10.096396 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:10.096212    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2" event={"ID":"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d","Type":"ContainerStarted","Data":"6f1fa95bd6e12880bfb282fd1c5ec1036fcc603e64582d19bb57c8d48b357472"}
Jul 07 03:01:11.102135 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:11.102099    2570 generic.go:356] "Generic (PLEG): container finished" podID="89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" containerID="c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11" exitCode=0
Jul 07 03:01:11.102551 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:11.102179    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2" event={"ID":"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d","Type":"ContainerDied","Data":"c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11"}
Jul 07 03:01:12.108645 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:12.108610    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2" event={"ID":"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d","Type":"ContainerStarted","Data":"70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf"}
Jul 07 03:01:12.109056 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:12.108729    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:01:12.129040 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:12.128997    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2" podStartSLOduration=2.493171879 podStartE2EDuration="3.128983444s" totalImagesPullingTime="0s" totalInitContainerRuntime="635.811565ms" isStatefulPod=false podCreationTimestamp="2026-07-07 03:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:01:12.127627945 +0000 UTC m=+2607.535217632" watchObservedRunningTime="2026-07-07 03:01:12.128983444 +0000 UTC m=+2607.536573130"
Jul 07 03:01:30.093209 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.093178    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"]
Jul 07 03:01:30.103019 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:01:30.102987    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:01:30.103019 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:01:30.103019 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:01:30.103019 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff" cmd=["/bin/sleep","15"]
Jul 07 03:01:30.103298 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.103032    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf" message="" err=<
Jul 07 03:01:30.103298 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:01:30.103298 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:01:30.103298 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:01:30.103298 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:01:30.103045    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:01:30.103298 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:01:30.103298 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:01:30.103298 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf" podUID="8fa1e220-6c9c-402b-8517-f3301e4ed00c" containerName="main" containerID="cri-o://288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff"
Jul 07 03:01:30.103298 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.103081    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf" podUID="8fa1e220-6c9c-402b-8517-f3301e4ed00c" containerName="main" containerID="cri-o://288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff" gracePeriod=60
Jul 07 03:01:30.356792 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.356725    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 03:01:30.402874 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.402845    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tokenizer-uds\") pod \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") "
Jul 07 03:01:30.403014 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.402952    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tls-certs\") pod \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") "
Jul 07 03:01:30.403014 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.403004    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwj4g\" (UniqueName: \"kubernetes.io/projected/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kube-api-access-rwj4g\") pod \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") "
Jul 07 03:01:30.403143 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.403087    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kserve-provision-location\") pod \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\" (UID: \"8fa1e220-6c9c-402b-8517-f3301e4ed00c\") "
Jul 07 03:01:30.403143 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.403089    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8fa1e220-6c9c-402b-8517-f3301e4ed00c" (UID: "8fa1e220-6c9c-402b-8517-f3301e4ed00c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:30.403289 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.403275    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:01:30.403947 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.403914    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fa1e220-6c9c-402b-8517-f3301e4ed00c" (UID: "8fa1e220-6c9c-402b-8517-f3301e4ed00c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:30.405159 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.405129    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kube-api-access-rwj4g" (OuterVolumeSpecName: "kube-api-access-rwj4g") pod "8fa1e220-6c9c-402b-8517-f3301e4ed00c" (UID: "8fa1e220-6c9c-402b-8517-f3301e4ed00c"). InnerVolumeSpecName "kube-api-access-rwj4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:01:30.405254 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.405177    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fa1e220-6c9c-402b-8517-f3301e4ed00c" (UID: "8fa1e220-6c9c-402b-8517-f3301e4ed00c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:01:30.503884 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.503857    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:01:30.503884 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.503882    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa1e220-6c9c-402b-8517-f3301e4ed00c-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:01:30.504015 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:30.503892    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwj4g\" (UniqueName: \"kubernetes.io/projected/8fa1e220-6c9c-402b-8517-f3301e4ed00c-kube-api-access-rwj4g\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:01:31.202476 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.202441    2570 generic.go:356] "Generic (PLEG): container finished" podID="8fa1e220-6c9c-402b-8517-f3301e4ed00c" containerID="288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff" exitCode=0
Jul 07 03:01:31.202872 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.202481    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf" event={"ID":"8fa1e220-6c9c-402b-8517-f3301e4ed00c","Type":"ContainerDied","Data":"288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff"}
Jul 07 03:01:31.202872 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.202530    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf" event={"ID":"8fa1e220-6c9c-402b-8517-f3301e4ed00c","Type":"ContainerDied","Data":"b69c97f1c7ec3262be13470f5ac5789c6f11a35bdb30a2d3b3f6b16e457d1e24"}
Jul 07 03:01:31.202872 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.202492    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"
Jul 07 03:01:31.202872 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.202551    2570 scope.go:117] "RemoveContainer" containerID="288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff"
Jul 07 03:01:31.211264 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.211244    2570 scope.go:117] "RemoveContainer" containerID="e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a"
Jul 07 03:01:31.218799 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.218781    2570 scope.go:117] "RemoveContainer" containerID="288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff"
Jul 07 03:01:31.219058 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:01:31.219041    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff\": container with ID starting with 288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff not found: ID does not exist" containerID="288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff"
Jul 07 03:01:31.219112 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.219065    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff"} err="failed to get container status \"288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff\": rpc error: code = NotFound desc = could not find container \"288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff\": container with ID starting with 288cceeab3337adb7d01b38f5bbf53833053b2d92fb0cb7b999ad1cc7f52edff not found: ID does not exist"
Jul 07 03:01:31.219112 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.219080    2570 scope.go:117] "RemoveContainer" containerID="e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a"
Jul 07 03:01:31.219429 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:01:31.219400    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a\": container with ID starting with e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a not found: ID does not exist" containerID="e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a"
Jul 07 03:01:31.219569 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.219439    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a"} err="failed to get container status \"e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a\": rpc error: code = NotFound desc = could not find container \"e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a\": container with ID starting with e401e2a744145dbf1adeae4fd4f94fc8bc03e0784ee0ca0ab28fc8f3f810dd3a not found: ID does not exist"
Jul 07 03:01:31.221209 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.221186    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"]
Jul 07 03:01:31.227311 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:31.227292    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-76bbb8774d-f4vbf"]
Jul 07 03:01:33.087053 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:33.087021    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa1e220-6c9c-402b-8517-f3301e4ed00c" path="/var/lib/kubelet/pods/8fa1e220-6c9c-402b-8517-f3301e4ed00c/volumes"
Jul 07 03:01:43.117050 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:01:43.117019    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:04:13.558217 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.558177    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"]
Jul 07 03:04:13.569097 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:04:13.569063    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:04:13.569097 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:04:13.569097 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:04:13.569097 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf" cmd=["/bin/sleep","15"]
Jul 07 03:04:13.569336 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.569112    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2" message="" err=<
Jul 07 03:04:13.569336 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:04:13.569336 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:04:13.569336 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:04:13.569336 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:04:13.569126    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:04:13.569336 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:04:13.569336 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:04:13.569336 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2" podUID="89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" containerName="main" containerID="cri-o://70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf"
Jul 07 03:04:13.569336 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.569171    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2" podUID="89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" containerName="main" containerID="cri-o://70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf" gracePeriod=60
Jul 07 03:04:13.828020 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.827998    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:04:13.892404 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.892378    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kserve-provision-location\") pod \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") "
Jul 07 03:04:13.892538 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.892421    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tls-certs\") pod \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") "
Jul 07 03:04:13.892538 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.892460    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tokenizer-uds\") pod \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") "
Jul 07 03:04:13.892538 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.892483    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndbb4\" (UniqueName: \"kubernetes.io/projected/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kube-api-access-ndbb4\") pod \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\" (UID: \"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d\") "
Jul 07 03:04:13.892731 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.892708    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" (UID: "89cedec9-11a2-4fc9-9b1f-7ba2c322f86d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:13.893010 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.892985    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" (UID: "89cedec9-11a2-4fc9-9b1f-7ba2c322f86d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:13.894519 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.894498    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" (UID: "89cedec9-11a2-4fc9-9b1f-7ba2c322f86d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:04:13.894614 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.894593    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kube-api-access-ndbb4" (OuterVolumeSpecName: "kube-api-access-ndbb4") pod "89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" (UID: "89cedec9-11a2-4fc9-9b1f-7ba2c322f86d"). InnerVolumeSpecName "kube-api-access-ndbb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:04:13.993598 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.993563    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:04:13.993598 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.993589    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndbb4\" (UniqueName: \"kubernetes.io/projected/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kube-api-access-ndbb4\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:04:13.993728 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.993612    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:04:13.993728 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:13.993626    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:04:14.004093 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.004063    2570 generic.go:356] "Generic (PLEG): container finished" podID="89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" containerID="70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf" exitCode=0
Jul 07 03:04:14.004194 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.004118    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"
Jul 07 03:04:14.004194 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.004130    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2" event={"ID":"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d","Type":"ContainerDied","Data":"70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf"}
Jul 07 03:04:14.004194 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.004188    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2" event={"ID":"89cedec9-11a2-4fc9-9b1f-7ba2c322f86d","Type":"ContainerDied","Data":"6f1fa95bd6e12880bfb282fd1c5ec1036fcc603e64582d19bb57c8d48b357472"}
Jul 07 03:04:14.004319 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.004218    2570 scope.go:117] "RemoveContainer" containerID="70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf"
Jul 07 03:04:14.013261 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.013243    2570 scope.go:117] "RemoveContainer" containerID="c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11"
Jul 07 03:04:14.020848 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.020829    2570 scope.go:117] "RemoveContainer" containerID="70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf"
Jul 07 03:04:14.021059 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:04:14.021043    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf\": container with ID starting with 70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf not found: ID does not exist" containerID="70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf"
Jul 07 03:04:14.021107 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.021066    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf"} err="failed to get container status \"70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf\": rpc error: code = NotFound desc = could not find container \"70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf\": container with ID starting with 70a0920017ffb77c89b89c35be58dff2e10fe5c6d30d82c8a1db6ec115b4fabf not found: ID does not exist"
Jul 07 03:04:14.021107 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.021082    2570 scope.go:117] "RemoveContainer" containerID="c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11"
Jul 07 03:04:14.021364 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:04:14.021331    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11\": container with ID starting with c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11 not found: ID does not exist" containerID="c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11"
Jul 07 03:04:14.021444 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.021373    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11"} err="failed to get container status \"c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11\": rpc error: code = NotFound desc = could not find container \"c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11\": container with ID starting with c421595fd86d636d177312bbde6b9b37557e5b46b759e271a5241f2fafad1d11 not found: ID does not exist"
Jul 07 03:04:14.025393 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.025372    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"]
Jul 07 03:04:14.031036 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:14.031017    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8579f4fd54-skhz2"]
Jul 07 03:04:15.086924 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:04:15.086883    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" path="/var/lib/kubelet/pods/89cedec9-11a2-4fc9-9b1f-7ba2c322f86d/volumes"
Jul 07 03:05:10.500529 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.500481    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"]
Jul 07 03:05:10.501046 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.501027    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" containerName="storage-initializer"
Jul 07 03:05:10.501098 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.501051    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" containerName="storage-initializer"
Jul 07 03:05:10.501098 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.501095    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" containerName="main"
Jul 07 03:05:10.501169 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.501104    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" containerName="main"
Jul 07 03:05:10.501169 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.501117    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa1e220-6c9c-402b-8517-f3301e4ed00c" containerName="storage-initializer"
Jul 07 03:05:10.501169 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.501127    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa1e220-6c9c-402b-8517-f3301e4ed00c" containerName="storage-initializer"
Jul 07 03:05:10.501169 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.501141    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa1e220-6c9c-402b-8517-f3301e4ed00c" containerName="main"
Jul 07 03:05:10.501169 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.501149    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa1e220-6c9c-402b-8517-f3301e4ed00c" containerName="main"
Jul 07 03:05:10.501324 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.501243    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="89cedec9-11a2-4fc9-9b1f-7ba2c322f86d" containerName="main"
Jul 07 03:05:10.501324 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.501256    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa1e220-6c9c-402b-8517-f3301e4ed00c" containerName="main"
Jul 07 03:05:10.505051 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.505028    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.507822 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.507797    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 03:05:10.508634 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.508611    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 07 03:05:10.508634 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.508625    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 03:05:10.508831 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.508614    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 03:05:10.508831 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.508616    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-pf7xn\""
Jul 07 03:05:10.509013 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.508991    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 03:05:10.515857 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.515838    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"]
Jul 07 03:05:10.638748 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.638725    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.638843 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.638777    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbkc\" (UniqueName: \"kubernetes.io/projected/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kube-api-access-6sbkc\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.638843 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.638798    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.638914 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.638889    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.739704 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.739673    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.739855 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.739716    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbkc\" (UniqueName: \"kubernetes.io/projected/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kube-api-access-6sbkc\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.739855 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.739734    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.739855 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.739800    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.740156 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.740128    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.740281 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.740177    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.742246 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.742224    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.749061 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.749040    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbkc\" (UniqueName: \"kubernetes.io/projected/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kube-api-access-6sbkc\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.815786 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.815765    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:10.950670 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:10.950636    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"]
Jul 07 03:05:10.953734 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:05:10.953704    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd5e52a_150f_4bc1_b638_15e2b5c8257e.slice/crio-2c55c6e738c45d5662ca44e3091c57ae75bf2013fa6c4cfa3a4aa92d772b8195 WatchSource:0}: Error finding container 2c55c6e738c45d5662ca44e3091c57ae75bf2013fa6c4cfa3a4aa92d772b8195: Status 404 returned error can't find the container with id 2c55c6e738c45d5662ca44e3091c57ae75bf2013fa6c4cfa3a4aa92d772b8195
Jul 07 03:05:11.276046 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:11.275998    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224" event={"ID":"3bd5e52a-150f-4bc1-b638-15e2b5c8257e","Type":"ContainerStarted","Data":"600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28"}
Jul 07 03:05:11.276206 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:11.276078    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224" event={"ID":"3bd5e52a-150f-4bc1-b638-15e2b5c8257e","Type":"ContainerStarted","Data":"2c55c6e738c45d5662ca44e3091c57ae75bf2013fa6c4cfa3a4aa92d772b8195"}
Jul 07 03:05:12.282737 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:12.282703    2570 generic.go:356] "Generic (PLEG): container finished" podID="3bd5e52a-150f-4bc1-b638-15e2b5c8257e" containerID="600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28" exitCode=0
Jul 07 03:05:12.283177 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:12.282793    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224" event={"ID":"3bd5e52a-150f-4bc1-b638-15e2b5c8257e","Type":"ContainerDied","Data":"600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28"}
Jul 07 03:05:13.288990 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:13.288957    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224" event={"ID":"3bd5e52a-150f-4bc1-b638-15e2b5c8257e","Type":"ContainerStarted","Data":"64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809"}
Jul 07 03:05:13.289391 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:13.289039    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:05:13.314976 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:13.314922    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224" podStartSLOduration=2.672964494 podStartE2EDuration="3.314905887s" totalImagesPullingTime="0s" totalInitContainerRuntime="641.941393ms" isStatefulPod=false podCreationTimestamp="2026-07-07 03:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:05:13.313390854 +0000 UTC m=+2848.720980543" watchObservedRunningTime="2026-07-07 03:05:13.314905887 +0000 UTC m=+2848.722495575"
Jul 07 03:05:44.295695 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:05:44.295663    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:07:28.201411 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.201375    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"]
Jul 07 03:07:28.204957 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.204942    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.209272 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.209232    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 07 03:07:28.209904 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.209865    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-epp-sa-dockercfg-cr62h\""
Jul 07 03:07:28.215898 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.215870    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d46pf\" (UniqueName: \"kubernetes.io/projected/5018a8ba-f0ba-43c9-b46b-509d581ea894-kube-api-access-d46pf\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.216008 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.215989    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.216065 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.216052    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5018a8ba-f0ba-43c9-b46b-509d581ea894-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.216176 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.216160    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.216471 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.216448    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"]
Jul 07 03:07:28.316656 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.316622    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.316805 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.316682    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d46pf\" (UniqueName: \"kubernetes.io/projected/5018a8ba-f0ba-43c9-b46b-509d581ea894-kube-api-access-d46pf\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.316805 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.316736    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.316805 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.316780    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5018a8ba-f0ba-43c9-b46b-509d581ea894-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.317066 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.317047    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.317111 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.317078    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.319394 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.319364    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5018a8ba-f0ba-43c9-b46b-509d581ea894-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.324230 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.324204    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d46pf\" (UniqueName: \"kubernetes.io/projected/5018a8ba-f0ba-43c9-b46b-509d581ea894-kube-api-access-d46pf\") pod \"tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.519268 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.519194    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:28.656476 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.656432    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"]
Jul 07 03:07:28.659261 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:07:28.659233    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5018a8ba_f0ba_43c9_b46b_509d581ea894.slice/crio-047fbab4ad6c7ec992ff1eaedc84adb50b9aa4537abbdf2718d5483d17c4d39c WatchSource:0}: Error finding container 047fbab4ad6c7ec992ff1eaedc84adb50b9aa4537abbdf2718d5483d17c4d39c: Status 404 returned error can't find the container with id 047fbab4ad6c7ec992ff1eaedc84adb50b9aa4537abbdf2718d5483d17c4d39c
Jul 07 03:07:28.947087 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.947054    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4" event={"ID":"5018a8ba-f0ba-43c9-b46b-509d581ea894","Type":"ContainerStarted","Data":"7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001"}
Jul 07 03:07:28.947272 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:28.947112    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4" event={"ID":"5018a8ba-f0ba-43c9-b46b-509d581ea894","Type":"ContainerStarted","Data":"047fbab4ad6c7ec992ff1eaedc84adb50b9aa4537abbdf2718d5483d17c4d39c"}
Jul 07 03:07:29.953474 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:29.953439    2570 generic.go:356] "Generic (PLEG): container finished" podID="5018a8ba-f0ba-43c9-b46b-509d581ea894" containerID="7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001" exitCode=0
Jul 07 03:07:29.953798 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:29.953533    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4" event={"ID":"5018a8ba-f0ba-43c9-b46b-509d581ea894","Type":"ContainerDied","Data":"7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001"}
Jul 07 03:07:30.962883 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:30.962847    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4" event={"ID":"5018a8ba-f0ba-43c9-b46b-509d581ea894","Type":"ContainerStarted","Data":"c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf"}
Jul 07 03:07:30.963293 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:30.962981    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:07:30.981802 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:07:30.981754    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4" podStartSLOduration=1.9244512980000001 podStartE2EDuration="2.981739191s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.057287893s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:07:30.979334923 +0000 UTC m=+2986.386924623" watchObservedRunningTime="2026-07-07 03:07:30.981739191 +0000 UTC m=+2986.389328887"
Jul 07 03:08:01.969924 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:01.969893    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:08:20.768318 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:20.768279    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"]
Jul 07 03:08:20.778656 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:08:20.778615    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:08:20.778656 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:08:20.778656 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:08:20.778656 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809" cmd=["/bin/sleep","15"]
Jul 07 03:08:20.778895 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:20.778676    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224" message="" err=<
Jul 07 03:08:20.778895 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:08:20.778895 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:08:20.778895 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:08:20.778895 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:08:20.778694    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:08:20.778895 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:08:20.778895 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:08:20.778895 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224" podUID="3bd5e52a-150f-4bc1-b638-15e2b5c8257e" containerName="main" containerID="cri-o://64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809"
Jul 07 03:08:20.778895 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:20.778752    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224" podUID="3bd5e52a-150f-4bc1-b638-15e2b5c8257e" containerName="main" containerID="cri-o://64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809" gracePeriod=60
Jul 07 03:08:21.033558 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.033531    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:08:21.149582 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.149553    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kserve-provision-location\") pod \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") "
Jul 07 03:08:21.149753 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.149614    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tls-certs\") pod \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") "
Jul 07 03:08:21.149753 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.149665    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sbkc\" (UniqueName: \"kubernetes.io/projected/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kube-api-access-6sbkc\") pod \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") "
Jul 07 03:08:21.149753 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.149703    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tokenizer-uds\") pod \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\" (UID: \"3bd5e52a-150f-4bc1-b638-15e2b5c8257e\") "
Jul 07 03:08:21.150049 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.150020    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3bd5e52a-150f-4bc1-b638-15e2b5c8257e" (UID: "3bd5e52a-150f-4bc1-b638-15e2b5c8257e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:21.150363 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.150278    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3bd5e52a-150f-4bc1-b638-15e2b5c8257e" (UID: "3bd5e52a-150f-4bc1-b638-15e2b5c8257e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:21.151870 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.151848    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3bd5e52a-150f-4bc1-b638-15e2b5c8257e" (UID: "3bd5e52a-150f-4bc1-b638-15e2b5c8257e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:08:21.151952 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.151903    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kube-api-access-6sbkc" (OuterVolumeSpecName: "kube-api-access-6sbkc") pod "3bd5e52a-150f-4bc1-b638-15e2b5c8257e" (UID: "3bd5e52a-150f-4bc1-b638-15e2b5c8257e"). InnerVolumeSpecName "kube-api-access-6sbkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:08:21.203381 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.203337    2570 generic.go:356] "Generic (PLEG): container finished" podID="3bd5e52a-150f-4bc1-b638-15e2b5c8257e" containerID="64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809" exitCode=0
Jul 07 03:08:21.203480 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.203396    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"
Jul 07 03:08:21.203480 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.203440    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224" event={"ID":"3bd5e52a-150f-4bc1-b638-15e2b5c8257e","Type":"ContainerDied","Data":"64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809"}
Jul 07 03:08:21.203575 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.203489    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224" event={"ID":"3bd5e52a-150f-4bc1-b638-15e2b5c8257e","Type":"ContainerDied","Data":"2c55c6e738c45d5662ca44e3091c57ae75bf2013fa6c4cfa3a4aa92d772b8195"}
Jul 07 03:08:21.203575 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.203511    2570 scope.go:117] "RemoveContainer" containerID="64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809"
Jul 07 03:08:21.212733 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.212715    2570 scope.go:117] "RemoveContainer" containerID="600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28"
Jul 07 03:08:21.220306 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.220289    2570 scope.go:117] "RemoveContainer" containerID="64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809"
Jul 07 03:08:21.220591 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:08:21.220570    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809\": container with ID starting with 64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809 not found: ID does not exist" containerID="64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809"
Jul 07 03:08:21.220678 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.220598    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809"} err="failed to get container status \"64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809\": rpc error: code = NotFound desc = could not find container \"64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809\": container with ID starting with 64948fee67797dbccbed394af69b0386c9b53c59b908fd787d02c73a28e88809 not found: ID does not exist"
Jul 07 03:08:21.220678 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.220617    2570 scope.go:117] "RemoveContainer" containerID="600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28"
Jul 07 03:08:21.220853 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:08:21.220832    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28\": container with ID starting with 600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28 not found: ID does not exist" containerID="600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28"
Jul 07 03:08:21.220908 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.220862    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28"} err="failed to get container status \"600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28\": rpc error: code = NotFound desc = could not find container \"600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28\": container with ID starting with 600f2302c2a54732163b00a0c6bed5cc196cf9313cc57159010d178773268b28 not found: ID does not exist"
Jul 07 03:08:21.224622 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.224601    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"]
Jul 07 03:08:21.228213 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.228195    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-sche5n224"]
Jul 07 03:08:21.250659 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.250644    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:08:21.250731 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.250663    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sbkc\" (UniqueName: \"kubernetes.io/projected/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kube-api-access-6sbkc\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:08:21.250731 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.250674    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:08:21.250731 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:21.250684    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3bd5e52a-150f-4bc1-b638-15e2b5c8257e-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:08:23.086982 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:23.086942    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bd5e52a-150f-4bc1-b638-15e2b5c8257e" path="/var/lib/kubelet/pods/3bd5e52a-150f-4bc1-b638-15e2b5c8257e/volumes"
Jul 07 03:08:47.733391 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.733341    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"]
Jul 07 03:08:47.734042 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.734019    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd5e52a-150f-4bc1-b638-15e2b5c8257e" containerName="storage-initializer"
Jul 07 03:08:47.734139 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.734047    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd5e52a-150f-4bc1-b638-15e2b5c8257e" containerName="storage-initializer"
Jul 07 03:08:47.734139 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.734085    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd5e52a-150f-4bc1-b638-15e2b5c8257e" containerName="main"
Jul 07 03:08:47.734139 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.734094    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd5e52a-150f-4bc1-b638-15e2b5c8257e" containerName="main"
Jul 07 03:08:47.734235 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.734179    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd5e52a-150f-4bc1-b638-15e2b5c8257e" containerName="main"
Jul 07 03:08:47.739399 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.739376    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.743025 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.743001    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-7lnw6\""
Jul 07 03:08:47.743136 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.743114    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 07 03:08:47.752708 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.752689    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"]
Jul 07 03:08:47.848151 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.848126    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.848258 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.848180    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.848304 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.848267    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.848340 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.848317    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4fqh\" (UniqueName: \"kubernetes.io/projected/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kube-api-access-m4fqh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.949586 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.949566    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.949696 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.949615    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4fqh\" (UniqueName: \"kubernetes.io/projected/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kube-api-access-m4fqh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.949696 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.949644    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.949696 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.949673    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.949942 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.949922    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.950010 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.949946    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.952061 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.952040    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:47.957577 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:47.957559    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4fqh\" (UniqueName: \"kubernetes.io/projected/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kube-api-access-m4fqh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:48.049724 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:48.049699    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:48.190848 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:48.190821    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"]
Jul 07 03:08:48.193041 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:08:48.193006    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426a9bd6_bedc_49d7_8fe0_e159ddddfe77.slice/crio-3c21a3d5cb3063ef939e43ceaabe766e9d4f4f9a5776bba8f0d8cb8a4ea13b0d WatchSource:0}: Error finding container 3c21a3d5cb3063ef939e43ceaabe766e9d4f4f9a5776bba8f0d8cb8a4ea13b0d: Status 404 returned error can't find the container with id 3c21a3d5cb3063ef939e43ceaabe766e9d4f4f9a5776bba8f0d8cb8a4ea13b0d
Jul 07 03:08:48.343824 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:48.343748    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j" event={"ID":"426a9bd6-bedc-49d7-8fe0-e159ddddfe77","Type":"ContainerStarted","Data":"6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4"}
Jul 07 03:08:48.343824 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:48.343796    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j" event={"ID":"426a9bd6-bedc-49d7-8fe0-e159ddddfe77","Type":"ContainerStarted","Data":"3c21a3d5cb3063ef939e43ceaabe766e9d4f4f9a5776bba8f0d8cb8a4ea13b0d"}
Jul 07 03:08:49.350339 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:49.350305    2570 generic.go:356] "Generic (PLEG): container finished" podID="426a9bd6-bedc-49d7-8fe0-e159ddddfe77" containerID="6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4" exitCode=0
Jul 07 03:08:49.350758 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:49.350403    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j" event={"ID":"426a9bd6-bedc-49d7-8fe0-e159ddddfe77","Type":"ContainerDied","Data":"6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4"}
Jul 07 03:08:50.358428 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:50.358392    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j" event={"ID":"426a9bd6-bedc-49d7-8fe0-e159ddddfe77","Type":"ContainerStarted","Data":"bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c"}
Jul 07 03:08:50.358812 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:50.358571    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:08:50.378316 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:08:50.378275    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j" podStartSLOduration=2.734325707 podStartE2EDuration="3.378262274s" totalImagesPullingTime="0s" totalInitContainerRuntime="643.936567ms" isStatefulPod=false podCreationTimestamp="2026-07-07 03:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:08:50.377439572 +0000 UTC m=+3065.785029260" watchObservedRunningTime="2026-07-07 03:08:50.378262274 +0000 UTC m=+3065.785851961"
Jul 07 03:09:21.364880 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:21.364851    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:09:43.578665 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:43.578631    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"]
Jul 07 03:09:43.588385 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:09:43.588333    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:09:43.588385 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:09:43.588385 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:09:43.588385 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf" cmd=["/bin/sleep","15"]
Jul 07 03:09:43.588676 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:43.588404    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4" message="" err=<
Jul 07 03:09:43.588676 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:09:43.588676 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:09:43.588676 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:09:43.588676 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:09:43.588422    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:09:43.588676 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:09:43.588676 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:09:43.588676 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4" podUID="5018a8ba-f0ba-43c9-b46b-509d581ea894" containerName="main" containerID="cri-o://c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf"
Jul 07 03:09:43.588676 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:43.588467    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4" podUID="5018a8ba-f0ba-43c9-b46b-509d581ea894" containerName="main" containerID="cri-o://c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf" gracePeriod=60
Jul 07 03:09:43.845181 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:43.845152    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:09:44.004211 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.004185    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d46pf\" (UniqueName: \"kubernetes.io/projected/5018a8ba-f0ba-43c9-b46b-509d581ea894-kube-api-access-d46pf\") pod \"5018a8ba-f0ba-43c9-b46b-509d581ea894\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") "
Jul 07 03:09:44.004388 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.004248    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-tokenizer-uds\") pod \"5018a8ba-f0ba-43c9-b46b-509d581ea894\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") "
Jul 07 03:09:44.004388 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.004270    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5018a8ba-f0ba-43c9-b46b-509d581ea894-tls-certs\") pod \"5018a8ba-f0ba-43c9-b46b-509d581ea894\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") "
Jul 07 03:09:44.004388 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.004300    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-kserve-provision-location\") pod \"5018a8ba-f0ba-43c9-b46b-509d581ea894\" (UID: \"5018a8ba-f0ba-43c9-b46b-509d581ea894\") "
Jul 07 03:09:44.004611 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.004587    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "5018a8ba-f0ba-43c9-b46b-509d581ea894" (UID: "5018a8ba-f0ba-43c9-b46b-509d581ea894"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:09:44.005018 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.005000    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5018a8ba-f0ba-43c9-b46b-509d581ea894" (UID: "5018a8ba-f0ba-43c9-b46b-509d581ea894"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:09:44.006494 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.006472    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5018a8ba-f0ba-43c9-b46b-509d581ea894-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5018a8ba-f0ba-43c9-b46b-509d581ea894" (UID: "5018a8ba-f0ba-43c9-b46b-509d581ea894"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:09:44.006543 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.006471    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5018a8ba-f0ba-43c9-b46b-509d581ea894-kube-api-access-d46pf" (OuterVolumeSpecName: "kube-api-access-d46pf") pod "5018a8ba-f0ba-43c9-b46b-509d581ea894" (UID: "5018a8ba-f0ba-43c9-b46b-509d581ea894"). InnerVolumeSpecName "kube-api-access-d46pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:09:44.104991 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.104925    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d46pf\" (UniqueName: \"kubernetes.io/projected/5018a8ba-f0ba-43c9-b46b-509d581ea894-kube-api-access-d46pf\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:09:44.104991 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.104955    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:09:44.104991 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.104964    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5018a8ba-f0ba-43c9-b46b-509d581ea894-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:09:44.104991 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.104972    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5018a8ba-f0ba-43c9-b46b-509d581ea894-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:09:44.627894 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.627862    2570 generic.go:356] "Generic (PLEG): container finished" podID="5018a8ba-f0ba-43c9-b46b-509d581ea894" containerID="c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf" exitCode=0
Jul 07 03:09:44.628387 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.627918    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"
Jul 07 03:09:44.628387 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.627933    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4" event={"ID":"5018a8ba-f0ba-43c9-b46b-509d581ea894","Type":"ContainerDied","Data":"c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf"}
Jul 07 03:09:44.628387 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.627982    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4" event={"ID":"5018a8ba-f0ba-43c9-b46b-509d581ea894","Type":"ContainerDied","Data":"047fbab4ad6c7ec992ff1eaedc84adb50b9aa4537abbdf2718d5483d17c4d39c"}
Jul 07 03:09:44.628387 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.628004    2570 scope.go:117] "RemoveContainer" containerID="c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf"
Jul 07 03:09:44.637773 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.637745    2570 scope.go:117] "RemoveContainer" containerID="7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001"
Jul 07 03:09:44.645748 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.645730    2570 scope.go:117] "RemoveContainer" containerID="c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf"
Jul 07 03:09:44.645971 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:09:44.645952    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf\": container with ID starting with c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf not found: ID does not exist" containerID="c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf"
Jul 07 03:09:44.646022 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.645978    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf"} err="failed to get container status \"c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf\": rpc error: code = NotFound desc = could not find container \"c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf\": container with ID starting with c31d1983f55da590d1468637a2b5cd4caac487090d8abe5ce5ca4e0cab1701cf not found: ID does not exist"
Jul 07 03:09:44.646022 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.645994    2570 scope.go:117] "RemoveContainer" containerID="7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001"
Jul 07 03:09:44.646185 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:09:44.646167    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001\": container with ID starting with 7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001 not found: ID does not exist" containerID="7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001"
Jul 07 03:09:44.646231 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.646187    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001"} err="failed to get container status \"7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001\": rpc error: code = NotFound desc = could not find container \"7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001\": container with ID starting with 7755ad98323a75ca759308fcec35fcbf09e6e4b880dedb83e753d34deb818001 not found: ID does not exist"
Jul 07 03:09:44.651019 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.650999    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"]
Jul 07 03:09:44.660589 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:44.660560    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-54fcb44dcb-ndbt4"]
Jul 07 03:09:45.086922 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:09:45.086892    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5018a8ba-f0ba-43c9-b46b-509d581ea894" path="/var/lib/kubelet/pods/5018a8ba-f0ba-43c9-b46b-509d581ea894/volumes"
Jul 07 03:10:09.509534 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.509459    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"]
Jul 07 03:10:09.509977 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.509956    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5018a8ba-f0ba-43c9-b46b-509d581ea894" containerName="main"
Jul 07 03:10:09.509977 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.509972    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5018a8ba-f0ba-43c9-b46b-509d581ea894" containerName="main"
Jul 07 03:10:09.510051 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.509983    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5018a8ba-f0ba-43c9-b46b-509d581ea894" containerName="storage-initializer"
Jul 07 03:10:09.510051 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.509988    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5018a8ba-f0ba-43c9-b46b-509d581ea894" containerName="storage-initializer"
Jul 07 03:10:09.510114 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.510062    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5018a8ba-f0ba-43c9-b46b-509d581ea894" containerName="main"
Jul 07 03:10:09.513173 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.513157    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.515790 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.515769    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-nl4nq\""
Jul 07 03:10:09.516621 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.516603    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 07 03:10:09.524634 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.524613    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"]
Jul 07 03:10:09.604369 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.604327    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c18ac232-3ba1-450d-8efe-cba5e67768a9-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.604470 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.604424    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.604470 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.604464    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xpn\" (UniqueName: \"kubernetes.io/projected/c18ac232-3ba1-450d-8efe-cba5e67768a9-kube-api-access-s8xpn\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.604548 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.604492    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.705748 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.705718    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xpn\" (UniqueName: \"kubernetes.io/projected/c18ac232-3ba1-450d-8efe-cba5e67768a9-kube-api-access-s8xpn\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.705872 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.705764    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.705872 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.705848    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c18ac232-3ba1-450d-8efe-cba5e67768a9-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.705980 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.705912    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.706196 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.706174    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.706272 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.706208    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.708527 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.708504    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c18ac232-3ba1-450d-8efe-cba5e67768a9-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.715716 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.715691    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xpn\" (UniqueName: \"kubernetes.io/projected/c18ac232-3ba1-450d-8efe-cba5e67768a9-kube-api-access-s8xpn\") pod \"prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.823090 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.823069    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:09.961825 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:09.961779    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"]
Jul 07 03:10:09.964033 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:10:09.964007    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18ac232_3ba1_450d_8efe_cba5e67768a9.slice/crio-6671499d742a1bd907d2b28a843d5b269deb1a1f144e9f7805ae8ca18342dd80 WatchSource:0}: Error finding container 6671499d742a1bd907d2b28a843d5b269deb1a1f144e9f7805ae8ca18342dd80: Status 404 returned error can't find the container with id 6671499d742a1bd907d2b28a843d5b269deb1a1f144e9f7805ae8ca18342dd80
Jul 07 03:10:10.758214 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:10.758188    2570 generic.go:356] "Generic (PLEG): container finished" podID="c18ac232-3ba1-450d-8efe-cba5e67768a9" containerID="a5a06a4ca6be46eb34421b0a0565fe59b82d148f8ccda966cc1abbd36cfc8b9e" exitCode=0
Jul 07 03:10:10.758548 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:10.758241    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx" event={"ID":"c18ac232-3ba1-450d-8efe-cba5e67768a9","Type":"ContainerDied","Data":"a5a06a4ca6be46eb34421b0a0565fe59b82d148f8ccda966cc1abbd36cfc8b9e"}
Jul 07 03:10:10.758548 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:10.758283    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx" event={"ID":"c18ac232-3ba1-450d-8efe-cba5e67768a9","Type":"ContainerStarted","Data":"6671499d742a1bd907d2b28a843d5b269deb1a1f144e9f7805ae8ca18342dd80"}
Jul 07 03:10:11.765103 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:11.765065    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx" event={"ID":"c18ac232-3ba1-450d-8efe-cba5e67768a9","Type":"ContainerStarted","Data":"aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a"}
Jul 07 03:10:11.765571 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:11.765157    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:10:11.783155 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:11.783100    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx" podStartSLOduration=2.105491894 podStartE2EDuration="2.783083141s" totalImagesPullingTime="0s" totalInitContainerRuntime="677.591247ms" isStatefulPod=false podCreationTimestamp="2026-07-07 03:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:10:11.782286112 +0000 UTC m=+3147.189875802" watchObservedRunningTime="2026-07-07 03:10:11.783083141 +0000 UTC m=+3147.190672829"
Jul 07 03:10:42.771742 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:10:42.771709    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:11:53.637386 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.637276    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"]
Jul 07 03:11:53.646727 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:11:53.646693    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:11:53.646727 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:11:53.646727 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:11:53.646727 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c" cmd=["/bin/sleep","15"]
Jul 07 03:11:53.647005 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.646741    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j" message="" err=<
Jul 07 03:11:53.647005 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:11:53.647005 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:11:53.647005 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:11:53.647005 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:11:53.646753    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:11:53.647005 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:11:53.647005 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:11:53.647005 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j" podUID="426a9bd6-bedc-49d7-8fe0-e159ddddfe77" containerName="main" containerID="cri-o://bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c"
Jul 07 03:11:53.647005 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.646797    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j" podUID="426a9bd6-bedc-49d7-8fe0-e159ddddfe77" containerName="main" containerID="cri-o://bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c" gracePeriod=60
Jul 07 03:11:53.898827 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.898770    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:11:53.950267 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.950240    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kserve-provision-location\") pod \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") "
Jul 07 03:11:53.950432 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.950280    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4fqh\" (UniqueName: \"kubernetes.io/projected/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kube-api-access-m4fqh\") pod \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") "
Jul 07 03:11:53.950432 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.950311    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tls-certs\") pod \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") "
Jul 07 03:11:53.950432 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.950338    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tokenizer-uds\") pod \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\" (UID: \"426a9bd6-bedc-49d7-8fe0-e159ddddfe77\") "
Jul 07 03:11:53.950700 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.950678    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "426a9bd6-bedc-49d7-8fe0-e159ddddfe77" (UID: "426a9bd6-bedc-49d7-8fe0-e159ddddfe77"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:11:53.951008 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.950982    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "426a9bd6-bedc-49d7-8fe0-e159ddddfe77" (UID: "426a9bd6-bedc-49d7-8fe0-e159ddddfe77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:11:53.952414 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.952395    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kube-api-access-m4fqh" (OuterVolumeSpecName: "kube-api-access-m4fqh") pod "426a9bd6-bedc-49d7-8fe0-e159ddddfe77" (UID: "426a9bd6-bedc-49d7-8fe0-e159ddddfe77"). InnerVolumeSpecName "kube-api-access-m4fqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:11:53.952414 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:53.952404    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "426a9bd6-bedc-49d7-8fe0-e159ddddfe77" (UID: "426a9bd6-bedc-49d7-8fe0-e159ddddfe77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:11:54.051422 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.051398    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:11:54.051422 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.051419    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:11:54.051550 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.051428    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4fqh\" (UniqueName: \"kubernetes.io/projected/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-kube-api-access-m4fqh\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:11:54.051550 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.051437    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426a9bd6-bedc-49d7-8fe0-e159ddddfe77-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:11:54.264462 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.264371    2570 generic.go:356] "Generic (PLEG): container finished" podID="426a9bd6-bedc-49d7-8fe0-e159ddddfe77" containerID="bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c" exitCode=0
Jul 07 03:11:54.264462 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.264419    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j" event={"ID":"426a9bd6-bedc-49d7-8fe0-e159ddddfe77","Type":"ContainerDied","Data":"bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c"}
Jul 07 03:11:54.264462 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.264457    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j" event={"ID":"426a9bd6-bedc-49d7-8fe0-e159ddddfe77","Type":"ContainerDied","Data":"3c21a3d5cb3063ef939e43ceaabe766e9d4f4f9a5776bba8f0d8cb8a4ea13b0d"}
Jul 07 03:11:54.264689 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.264477    2570 scope.go:117] "RemoveContainer" containerID="bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c"
Jul 07 03:11:54.264689 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.264422    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"
Jul 07 03:11:54.274167 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.274150    2570 scope.go:117] "RemoveContainer" containerID="6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4"
Jul 07 03:11:54.281847 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.281830    2570 scope.go:117] "RemoveContainer" containerID="bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c"
Jul 07 03:11:54.282095 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:11:54.282079    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c\": container with ID starting with bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c not found: ID does not exist" containerID="bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c"
Jul 07 03:11:54.282133 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.282104    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c"} err="failed to get container status \"bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c\": rpc error: code = NotFound desc = could not find container \"bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c\": container with ID starting with bc25aad990256827d36f61ed435a48b1883a4f7c64d7516210ba4b681f05001c not found: ID does not exist"
Jul 07 03:11:54.282133 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.282119    2570 scope.go:117] "RemoveContainer" containerID="6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4"
Jul 07 03:11:54.282357 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:11:54.282326    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4\": container with ID starting with 6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4 not found: ID does not exist" containerID="6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4"
Jul 07 03:11:54.282399 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.282370    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4"} err="failed to get container status \"6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4\": rpc error: code = NotFound desc = could not find container \"6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4\": container with ID starting with 6e63c730698de34430f593ab09627bb3872ec0e0fafea98432344875a37d67b4 not found: ID does not exist"
Jul 07 03:11:54.288463 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.288442    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"]
Jul 07 03:11:54.294838 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:54.294813    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-c7d44ckr2j"]
Jul 07 03:11:55.087194 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:11:55.087159    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426a9bd6-bedc-49d7-8fe0-e159ddddfe77" path="/var/lib/kubelet/pods/426a9bd6-bedc-49d7-8fe0-e159ddddfe77/volumes"
Jul 07 03:12:30.125047 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.125013    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"]
Jul 07 03:12:30.125499 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.125395    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426a9bd6-bedc-49d7-8fe0-e159ddddfe77" containerName="main"
Jul 07 03:12:30.125499 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.125407    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="426a9bd6-bedc-49d7-8fe0-e159ddddfe77" containerName="main"
Jul 07 03:12:30.125499 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.125431    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426a9bd6-bedc-49d7-8fe0-e159ddddfe77" containerName="storage-initializer"
Jul 07 03:12:30.125499 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.125437    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="426a9bd6-bedc-49d7-8fe0-e159ddddfe77" containerName="storage-initializer"
Jul 07 03:12:30.125499 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.125495    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="426a9bd6-bedc-49d7-8fe0-e159ddddfe77" containerName="main"
Jul 07 03:12:30.128562 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.128542    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.130990 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.130972    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 07 03:12:30.131084 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.131064    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-r2gb2\""
Jul 07 03:12:30.138550 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.138530    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"]
Jul 07 03:12:30.244457 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.244427    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.244581 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.244526    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.244631 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.244582    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfqqg\" (UniqueName: \"kubernetes.io/projected/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kube-api-access-pfqqg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.244667 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.244638    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.345703 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.345683    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.345811 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.345710    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfqqg\" (UniqueName: \"kubernetes.io/projected/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kube-api-access-pfqqg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.345811 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.345736    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.345811 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.345782    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.346036 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.346016    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.346110 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.346083    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.348404 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.348383    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.353969 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.353948    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfqqg\" (UniqueName: \"kubernetes.io/projected/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kube-api-access-pfqqg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.438455 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.438393    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:30.574709 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:30.574666    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"]
Jul 07 03:12:30.576067 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:12:30.576045    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe28397_19fa_4a5d_9d6c_ce0f5a15e351.slice/crio-5bba89968ceacbef0da4b562ded4bbb7406d20731d73aca09330728c1764e8da WatchSource:0}: Error finding container 5bba89968ceacbef0da4b562ded4bbb7406d20731d73aca09330728c1764e8da: Status 404 returned error can't find the container with id 5bba89968ceacbef0da4b562ded4bbb7406d20731d73aca09330728c1764e8da
Jul 07 03:12:31.452065 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:31.452032    2570 generic.go:356] "Generic (PLEG): container finished" podID="3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" containerID="06f6caa971ad05e3883475de2e197e804d0a45dd704983994c406ec88f72093a" exitCode=0
Jul 07 03:12:31.452416 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:31.452123    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx" event={"ID":"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351","Type":"ContainerDied","Data":"06f6caa971ad05e3883475de2e197e804d0a45dd704983994c406ec88f72093a"}
Jul 07 03:12:31.452416 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:31.452185    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx" event={"ID":"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351","Type":"ContainerStarted","Data":"5bba89968ceacbef0da4b562ded4bbb7406d20731d73aca09330728c1764e8da"}
Jul 07 03:12:32.458782 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:32.458749    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx" event={"ID":"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351","Type":"ContainerStarted","Data":"cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382"}
Jul 07 03:12:32.459147 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:32.458881    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:12:32.479361 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:32.479305    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx" podStartSLOduration=1.860837229 podStartE2EDuration="2.479292851s" totalImagesPullingTime="0s" totalInitContainerRuntime="618.455622ms" isStatefulPod=false podCreationTimestamp="2026-07-07 03:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:12:32.477834948 +0000 UTC m=+3287.885424659" watchObservedRunningTime="2026-07-07 03:12:32.479292851 +0000 UTC m=+3287.886882538"
Jul 07 03:12:52.978126 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:52.978092    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"]
Jul 07 03:12:52.987693 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:12:52.987652    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:12:52.987693 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:12:52.987693 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:12:52.987693 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a" cmd=["/bin/sleep","15"]
Jul 07 03:12:52.987962 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:52.987714    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx" message="" err=<
Jul 07 03:12:52.987962 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:12:52.987962 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:12:52.987962 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:12:52.987962 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:12:52.987729    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:12:52.987962 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:12:52.987962 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:12:52.987962 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx" podUID="c18ac232-3ba1-450d-8efe-cba5e67768a9" containerName="main" containerID="cri-o://aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a"
Jul 07 03:12:52.987962 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:52.987784    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx" podUID="c18ac232-3ba1-450d-8efe-cba5e67768a9" containerName="main" containerID="cri-o://aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a" gracePeriod=60
Jul 07 03:12:53.249727 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.249702    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:12:53.339859 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.339836    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c18ac232-3ba1-450d-8efe-cba5e67768a9-tls-certs\") pod \"c18ac232-3ba1-450d-8efe-cba5e67768a9\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") "
Jul 07 03:12:53.340017 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.339882    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-kserve-provision-location\") pod \"c18ac232-3ba1-450d-8efe-cba5e67768a9\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") "
Jul 07 03:12:53.340017 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.339906    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8xpn\" (UniqueName: \"kubernetes.io/projected/c18ac232-3ba1-450d-8efe-cba5e67768a9-kube-api-access-s8xpn\") pod \"c18ac232-3ba1-450d-8efe-cba5e67768a9\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") "
Jul 07 03:12:53.340017 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.339927    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-tokenizer-uds\") pod \"c18ac232-3ba1-450d-8efe-cba5e67768a9\" (UID: \"c18ac232-3ba1-450d-8efe-cba5e67768a9\") "
Jul 07 03:12:53.340227 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.340206    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c18ac232-3ba1-450d-8efe-cba5e67768a9" (UID: "c18ac232-3ba1-450d-8efe-cba5e67768a9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:53.340571 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.340545    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c18ac232-3ba1-450d-8efe-cba5e67768a9" (UID: "c18ac232-3ba1-450d-8efe-cba5e67768a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:53.342098 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.342071    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18ac232-3ba1-450d-8efe-cba5e67768a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c18ac232-3ba1-450d-8efe-cba5e67768a9" (UID: "c18ac232-3ba1-450d-8efe-cba5e67768a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:12:53.342098 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.342071    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18ac232-3ba1-450d-8efe-cba5e67768a9-kube-api-access-s8xpn" (OuterVolumeSpecName: "kube-api-access-s8xpn") pod "c18ac232-3ba1-450d-8efe-cba5e67768a9" (UID: "c18ac232-3ba1-450d-8efe-cba5e67768a9"). InnerVolumeSpecName "kube-api-access-s8xpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:12:53.441106 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.441073    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c18ac232-3ba1-450d-8efe-cba5e67768a9-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:12:53.441106 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.441100    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:12:53.441106 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.441111    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8xpn\" (UniqueName: \"kubernetes.io/projected/c18ac232-3ba1-450d-8efe-cba5e67768a9-kube-api-access-s8xpn\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:12:53.441323 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.441120    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c18ac232-3ba1-450d-8efe-cba5e67768a9-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:12:53.565389 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.565332    2570 generic.go:356] "Generic (PLEG): container finished" podID="c18ac232-3ba1-450d-8efe-cba5e67768a9" containerID="aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a" exitCode=0
Jul 07 03:12:53.565496 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.565419    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"
Jul 07 03:12:53.565496 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.565422    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx" event={"ID":"c18ac232-3ba1-450d-8efe-cba5e67768a9","Type":"ContainerDied","Data":"aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a"}
Jul 07 03:12:53.565496 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.565480    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx" event={"ID":"c18ac232-3ba1-450d-8efe-cba5e67768a9","Type":"ContainerDied","Data":"6671499d742a1bd907d2b28a843d5b269deb1a1f144e9f7805ae8ca18342dd80"}
Jul 07 03:12:53.565602 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.565502    2570 scope.go:117] "RemoveContainer" containerID="aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a"
Jul 07 03:12:53.575523 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.575508    2570 scope.go:117] "RemoveContainer" containerID="a5a06a4ca6be46eb34421b0a0565fe59b82d148f8ccda966cc1abbd36cfc8b9e"
Jul 07 03:12:53.583636 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.583618    2570 scope.go:117] "RemoveContainer" containerID="aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a"
Jul 07 03:12:53.583883 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:12:53.583865    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a\": container with ID starting with aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a not found: ID does not exist" containerID="aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a"
Jul 07 03:12:53.583937 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.583892    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a"} err="failed to get container status \"aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a\": rpc error: code = NotFound desc = could not find container \"aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a\": container with ID starting with aaa1a27c2f06d7b8324ba10143ce1a6be52624a631728420f983a71d1115b00a not found: ID does not exist"
Jul 07 03:12:53.583937 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.583910    2570 scope.go:117] "RemoveContainer" containerID="a5a06a4ca6be46eb34421b0a0565fe59b82d148f8ccda966cc1abbd36cfc8b9e"
Jul 07 03:12:53.584138 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:12:53.584120    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a06a4ca6be46eb34421b0a0565fe59b82d148f8ccda966cc1abbd36cfc8b9e\": container with ID starting with a5a06a4ca6be46eb34421b0a0565fe59b82d148f8ccda966cc1abbd36cfc8b9e not found: ID does not exist" containerID="a5a06a4ca6be46eb34421b0a0565fe59b82d148f8ccda966cc1abbd36cfc8b9e"
Jul 07 03:12:53.584191 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.584147    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a06a4ca6be46eb34421b0a0565fe59b82d148f8ccda966cc1abbd36cfc8b9e"} err="failed to get container status \"a5a06a4ca6be46eb34421b0a0565fe59b82d148f8ccda966cc1abbd36cfc8b9e\": rpc error: code = NotFound desc = could not find container \"a5a06a4ca6be46eb34421b0a0565fe59b82d148f8ccda966cc1abbd36cfc8b9e\": container with ID starting with a5a06a4ca6be46eb34421b0a0565fe59b82d148f8ccda966cc1abbd36cfc8b9e not found: ID does not exist"
Jul 07 03:12:53.586762 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.586742    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"]
Jul 07 03:12:53.590718 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:53.590700    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-66cbfd5b6d-g9qrx"]
Jul 07 03:12:55.086794 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:12:55.086762    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c18ac232-3ba1-450d-8efe-cba5e67768a9" path="/var/lib/kubelet/pods/c18ac232-3ba1-450d-8efe-cba5e67768a9/volumes"
Jul 07 03:13:03.466532 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:13:03.466502    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:14:04.143455 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.143421    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"]
Jul 07 03:14:04.144030 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.144013    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c18ac232-3ba1-450d-8efe-cba5e67768a9" containerName="main"
Jul 07 03:14:04.144075 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.144036    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c18ac232-3ba1-450d-8efe-cba5e67768a9" containerName="main"
Jul 07 03:14:04.144075 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.144049    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c18ac232-3ba1-450d-8efe-cba5e67768a9" containerName="storage-initializer"
Jul 07 03:14:04.144075 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.144059    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c18ac232-3ba1-450d-8efe-cba5e67768a9" containerName="storage-initializer"
Jul 07 03:14:04.144171 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.144160    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c18ac232-3ba1-450d-8efe-cba5e67768a9" containerName="main"
Jul 07 03:14:04.147808 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.147792    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.151241 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.151220    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 07 03:14:04.151338 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.151224    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-epp-sa-dockercfg-hzwmf\""
Jul 07 03:14:04.157310 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.157276    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"]
Jul 07 03:14:04.200937 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.200912    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.201034 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.200946    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2d9c\" (UniqueName: \"kubernetes.io/projected/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-kube-api-access-p2d9c\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.201089 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.201032    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.301664 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.301637    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.301778 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.301687    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.301778 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.301743    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2d9c\" (UniqueName: \"kubernetes.io/projected/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-kube-api-access-p2d9c\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.302000 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.301981    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.304090 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.304070    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.310888 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.310863    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2d9c\" (UniqueName: \"kubernetes.io/projected/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-kube-api-access-p2d9c\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.457441 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.457392    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.591786 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.591760    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"]
Jul 07 03:14:04.593748 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:14:04.593727    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16f9f48_a3f8_41ea_93f0_e906f6d95c45.slice/crio-c101489b5aa5edc53190d0d3da0aac63788c39a52d610761aa0e772bb1347b00 WatchSource:0}: Error finding container c101489b5aa5edc53190d0d3da0aac63788c39a52d610761aa0e772bb1347b00: Status 404 returned error can't find the container with id c101489b5aa5edc53190d0d3da0aac63788c39a52d610761aa0e772bb1347b00
Jul 07 03:14:04.933567 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.933538    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv" event={"ID":"e16f9f48-a3f8-41ea-93f0-e906f6d95c45","Type":"ContainerStarted","Data":"074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115"}
Jul 07 03:14:04.933761 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.933584    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv" event={"ID":"e16f9f48-a3f8-41ea-93f0-e906f6d95c45","Type":"ContainerStarted","Data":"c101489b5aa5edc53190d0d3da0aac63788c39a52d610761aa0e772bb1347b00"}
Jul 07 03:14:04.933761 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.933656    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:04.965552 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:04.965507    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv" podStartSLOduration=0.965491925 podStartE2EDuration="965.491925ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:14:04.964882251 +0000 UTC m=+3380.372471939" watchObservedRunningTime="2026-07-07 03:14:04.965491925 +0000 UTC m=+3380.373081612"
Jul 07 03:14:35.940193 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:35.940147    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:50.758366 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:50.753196    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"]
Jul 07 03:14:50.770621 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:14:50.770577    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:14:50.770621 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:14:50.770621 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:14:50.770621 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115" cmd=["/bin/sleep","15"]
Jul 07 03:14:50.770923 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:50.770642    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv" message="" err=<
Jul 07 03:14:50.770923 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:14:50.770923 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:14:50.770923 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:14:50.770923 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:14:50.770659    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:14:50.770923 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:14:50.770923 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:14:50.770923 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv" podUID="e16f9f48-a3f8-41ea-93f0-e906f6d95c45" containerName="main" containerID="cri-o://074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115"
Jul 07 03:14:50.770923 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:50.770730    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv" podUID="e16f9f48-a3f8-41ea-93f0-e906f6d95c45" containerName="main" containerID="cri-o://074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115" gracePeriod=60
Jul 07 03:14:51.014014 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.013957    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:51.078711 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.078683    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tokenizer-uds\") pod \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") "
Jul 07 03:14:51.078832 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.078724    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tls-certs\") pod \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") "
Jul 07 03:14:51.078832 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.078781    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2d9c\" (UniqueName: \"kubernetes.io/projected/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-kube-api-access-p2d9c\") pod \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\" (UID: \"e16f9f48-a3f8-41ea-93f0-e906f6d95c45\") "
Jul 07 03:14:51.078925 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.078903    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e16f9f48-a3f8-41ea-93f0-e906f6d95c45" (UID: "e16f9f48-a3f8-41ea-93f0-e906f6d95c45"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:14:51.080883 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.080862    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e16f9f48-a3f8-41ea-93f0-e906f6d95c45" (UID: "e16f9f48-a3f8-41ea-93f0-e906f6d95c45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:14:51.080979 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.080913    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-kube-api-access-p2d9c" (OuterVolumeSpecName: "kube-api-access-p2d9c") pod "e16f9f48-a3f8-41ea-93f0-e906f6d95c45" (UID: "e16f9f48-a3f8-41ea-93f0-e906f6d95c45"). InnerVolumeSpecName "kube-api-access-p2d9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:14:51.156715 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.156694    2570 generic.go:356] "Generic (PLEG): container finished" podID="e16f9f48-a3f8-41ea-93f0-e906f6d95c45" containerID="074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115" exitCode=0
Jul 07 03:14:51.156816 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.156731    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"
Jul 07 03:14:51.156816 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.156786    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv" event={"ID":"e16f9f48-a3f8-41ea-93f0-e906f6d95c45","Type":"ContainerDied","Data":"074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115"}
Jul 07 03:14:51.156945 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.156834    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv" event={"ID":"e16f9f48-a3f8-41ea-93f0-e906f6d95c45","Type":"ContainerDied","Data":"c101489b5aa5edc53190d0d3da0aac63788c39a52d610761aa0e772bb1347b00"}
Jul 07 03:14:51.156945 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.156865    2570 scope.go:117] "RemoveContainer" containerID="074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115"
Jul 07 03:14:51.169280 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.169264    2570 scope.go:117] "RemoveContainer" containerID="074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115"
Jul 07 03:14:51.169553 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:14:51.169538    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115\": container with ID starting with 074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115 not found: ID does not exist" containerID="074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115"
Jul 07 03:14:51.169604 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.169561    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115"} err="failed to get container status \"074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115\": rpc error: code = NotFound desc = could not find container \"074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115\": container with ID starting with 074bc49d42144389696766716958105c2d201bea92fbe991ae4000461f299115 not found: ID does not exist"
Jul 07 03:14:51.172173 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.172153    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"]
Jul 07 03:14:51.175796 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.175779    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-4wbfv"]
Jul 07 03:14:51.180230 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.180207    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2d9c\" (UniqueName: \"kubernetes.io/projected/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-kube-api-access-p2d9c\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:14:51.180230 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.180227    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:14:51.180398 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:51.180238    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e16f9f48-a3f8-41ea-93f0-e906f6d95c45-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:14:53.087289 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:14:53.087260    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e16f9f48-a3f8-41ea-93f0-e906f6d95c45" path="/var/lib/kubelet/pods/e16f9f48-a3f8-41ea-93f0-e906f6d95c45/volumes"
Jul 07 03:15:28.530857 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.530814    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"]
Jul 07 03:15:28.540633 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:15:28.540597    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:15:28.540633 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:15:28.540633 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:15:28.540633 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382" cmd=["/bin/sleep","15"]
Jul 07 03:15:28.540965 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.540646    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx" message="" err=<
Jul 07 03:15:28.540965 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:15:28.540965 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:15:28.540965 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:15:28.540965 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:15:28.540658    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:15:28.540965 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:15:28.540965 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:15:28.540965 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx" podUID="3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" containerName="main" containerID="cri-o://cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382"
Jul 07 03:15:28.540965 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.540701    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx" podUID="3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" containerName="main" containerID="cri-o://cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382" gracePeriod=60
Jul 07 03:15:28.788012 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.787960    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:15:28.881397 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.881372    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tokenizer-uds\") pod \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") "
Jul 07 03:15:28.881523 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.881416    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kserve-provision-location\") pod \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") "
Jul 07 03:15:28.881523 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.881440    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tls-certs\") pod \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") "
Jul 07 03:15:28.881523 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.881476    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfqqg\" (UniqueName: \"kubernetes.io/projected/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kube-api-access-pfqqg\") pod \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\" (UID: \"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351\") "
Jul 07 03:15:28.881665 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.881587    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" (UID: "3fe28397-19fa-4a5d-9d6c-ce0f5a15e351"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:28.881718 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.881705    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:15:28.882030 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.882009    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" (UID: "3fe28397-19fa-4a5d-9d6c-ce0f5a15e351"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:28.883566 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.883543    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" (UID: "3fe28397-19fa-4a5d-9d6c-ce0f5a15e351"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:15:28.883744 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.883726    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kube-api-access-pfqqg" (OuterVolumeSpecName: "kube-api-access-pfqqg") pod "3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" (UID: "3fe28397-19fa-4a5d-9d6c-ce0f5a15e351"). InnerVolumeSpecName "kube-api-access-pfqqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:15:28.982365 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.982320    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:15:28.982365 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.982361    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:15:28.982517 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:28.982372    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfqqg\" (UniqueName: \"kubernetes.io/projected/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351-kube-api-access-pfqqg\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:15:29.351855 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.351767    2570 generic.go:356] "Generic (PLEG): container finished" podID="3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" containerID="cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382" exitCode=0
Jul 07 03:15:29.351855 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.351822    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"
Jul 07 03:15:29.352036 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.351861    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx" event={"ID":"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351","Type":"ContainerDied","Data":"cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382"}
Jul 07 03:15:29.352036 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.351907    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx" event={"ID":"3fe28397-19fa-4a5d-9d6c-ce0f5a15e351","Type":"ContainerDied","Data":"5bba89968ceacbef0da4b562ded4bbb7406d20731d73aca09330728c1764e8da"}
Jul 07 03:15:29.352036 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.351927    2570 scope.go:117] "RemoveContainer" containerID="cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382"
Jul 07 03:15:29.361414 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.361397    2570 scope.go:117] "RemoveContainer" containerID="06f6caa971ad05e3883475de2e197e804d0a45dd704983994c406ec88f72093a"
Jul 07 03:15:29.367917 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.367893    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"]
Jul 07 03:15:29.369632 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.369565    2570 scope.go:117] "RemoveContainer" containerID="cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382"
Jul 07 03:15:29.369943 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:15:29.369910    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382\": container with ID starting with cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382 not found: ID does not exist" containerID="cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382"
Jul 07 03:15:29.370146 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.369954    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382"} err="failed to get container status \"cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382\": rpc error: code = NotFound desc = could not find container \"cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382\": container with ID starting with cc67124ae277d28d9309ef3f8b7c5492076f347b2b8c977b2f27f29102b25382 not found: ID does not exist"
Jul 07 03:15:29.370146 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.369978    2570 scope.go:117] "RemoveContainer" containerID="06f6caa971ad05e3883475de2e197e804d0a45dd704983994c406ec88f72093a"
Jul 07 03:15:29.370320 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:15:29.370293    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f6caa971ad05e3883475de2e197e804d0a45dd704983994c406ec88f72093a\": container with ID starting with 06f6caa971ad05e3883475de2e197e804d0a45dd704983994c406ec88f72093a not found: ID does not exist" containerID="06f6caa971ad05e3883475de2e197e804d0a45dd704983994c406ec88f72093a"
Jul 07 03:15:29.370418 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.370338    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f6caa971ad05e3883475de2e197e804d0a45dd704983994c406ec88f72093a"} err="failed to get container status \"06f6caa971ad05e3883475de2e197e804d0a45dd704983994c406ec88f72093a\": rpc error: code = NotFound desc = could not find container \"06f6caa971ad05e3883475de2e197e804d0a45dd704983994c406ec88f72093a\": container with ID starting with 06f6caa971ad05e3883475de2e197e804d0a45dd704983994c406ec88f72093a not found: ID does not exist"
Jul 07 03:15:29.371784 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:29.371754    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-b78dc9c7fwr2tx"]
Jul 07 03:15:31.086609 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:15:31.086579    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" path="/var/lib/kubelet/pods/3fe28397-19fa-4a5d-9d6c-ce0f5a15e351/volumes"
Jul 07 03:16:54.162490 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.162461    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"]
Jul 07 03:16:54.162895 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.162803    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" containerName="main"
Jul 07 03:16:54.162895 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.162815    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" containerName="main"
Jul 07 03:16:54.162895 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.162831    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" containerName="storage-initializer"
Jul 07 03:16:54.162895 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.162836    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" containerName="storage-initializer"
Jul 07 03:16:54.162895 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.162852    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16f9f48-a3f8-41ea-93f0-e906f6d95c45" containerName="main"
Jul 07 03:16:54.162895 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.162857    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16f9f48-a3f8-41ea-93f0-e906f6d95c45" containerName="main"
Jul 07 03:16:54.163082 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.162916    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16f9f48-a3f8-41ea-93f0-e906f6d95c45" containerName="main"
Jul 07 03:16:54.163082 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.162925    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe28397-19fa-4a5d-9d6c-ce0f5a15e351" containerName="main"
Jul 07 03:16:54.166002 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.165984    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.169280 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.169244    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 03:16:54.169443 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.169286    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 03:16:54.169443 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.169288    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 03:16:54.169443 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.169325    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 07 03:16:54.169443 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.169334    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-zpxxw\""
Jul 07 03:16:54.174574 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.174480    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"]
Jul 07 03:16:54.242291 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.242265    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedf6d41-af4e-471e-a60e-a6662648cd98-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.242436 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.242298    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh76l\" (UniqueName: \"kubernetes.io/projected/dedf6d41-af4e-471e-a60e-a6662648cd98-kube-api-access-jh76l\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.242436 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.242421    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dedf6d41-af4e-471e-a60e-a6662648cd98-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.343533 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.343496    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedf6d41-af4e-471e-a60e-a6662648cd98-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.343533 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.343537    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh76l\" (UniqueName: \"kubernetes.io/projected/dedf6d41-af4e-471e-a60e-a6662648cd98-kube-api-access-jh76l\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.343724 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.343585    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dedf6d41-af4e-471e-a60e-a6662648cd98-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.343961 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.343945    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dedf6d41-af4e-471e-a60e-a6662648cd98-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.346174 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.346142    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedf6d41-af4e-471e-a60e-a6662648cd98-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.351702 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.351677    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh76l\" (UniqueName: \"kubernetes.io/projected/dedf6d41-af4e-471e-a60e-a6662648cd98-kube-api-access-jh76l\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.477052 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.476982    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.609051 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.609026    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"]
Jul 07 03:16:54.611769 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:16:54.611743    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedf6d41_af4e_471e_a60e_a6662648cd98.slice/crio-71fb8c9a776d916744d380099bda9bb352db1fc781d0af48701eb9838b174540 WatchSource:0}: Error finding container 71fb8c9a776d916744d380099bda9bb352db1fc781d0af48701eb9838b174540: Status 404 returned error can't find the container with id 71fb8c9a776d916744d380099bda9bb352db1fc781d0af48701eb9838b174540
Jul 07 03:16:54.765667 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.765578    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb" event={"ID":"dedf6d41-af4e-471e-a60e-a6662648cd98","Type":"ContainerStarted","Data":"c6f3b55038b65962ffb423e08dbacdaacc74125341bdf756b2816c12a5648ba4"}
Jul 07 03:16:54.765667 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.765640    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb" event={"ID":"dedf6d41-af4e-471e-a60e-a6662648cd98","Type":"ContainerStarted","Data":"71fb8c9a776d916744d380099bda9bb352db1fc781d0af48701eb9838b174540"}
Jul 07 03:16:54.765667 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.765663    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:16:54.785472 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:16:54.785430    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb" podStartSLOduration=0.78541519 podStartE2EDuration="785.41519ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:16:54.783688501 +0000 UTC m=+3550.191278200" watchObservedRunningTime="2026-07-07 03:16:54.78541519 +0000 UTC m=+3550.193004877"
Jul 07 03:17:25.772396 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:25.772364    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:17:26.770142 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:26.770111    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"]
Jul 07 03:17:26.780488 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:17:26.780450    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:17:26.780488 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:17:26.780488 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:17:26.780488 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="c6f3b55038b65962ffb423e08dbacdaacc74125341bdf756b2816c12a5648ba4" cmd=["/bin/sleep","15"]
Jul 07 03:17:26.781053 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:26.780503    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb" message="" err=<
Jul 07 03:17:26.781053 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:17:26.781053 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:17:26.781053 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:17:26.781053 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:17:26.780516    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:17:26.781053 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:17:26.781053 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:17:26.781053 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb" podUID="dedf6d41-af4e-471e-a60e-a6662648cd98" containerName="main" containerID="cri-o://c6f3b55038b65962ffb423e08dbacdaacc74125341bdf756b2816c12a5648ba4"
Jul 07 03:17:26.781053 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:26.780556    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb" podUID="dedf6d41-af4e-471e-a60e-a6662648cd98" containerName="main" containerID="cri-o://c6f3b55038b65962ffb423e08dbacdaacc74125341bdf756b2816c12a5648ba4" gracePeriod=60
Jul 07 03:17:26.922111 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:26.922085    2570 generic.go:356] "Generic (PLEG): container finished" podID="dedf6d41-af4e-471e-a60e-a6662648cd98" containerID="c6f3b55038b65962ffb423e08dbacdaacc74125341bdf756b2816c12a5648ba4" exitCode=0
Jul 07 03:17:26.922236 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:26.922160    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb" event={"ID":"dedf6d41-af4e-471e-a60e-a6662648cd98","Type":"ContainerDied","Data":"c6f3b55038b65962ffb423e08dbacdaacc74125341bdf756b2816c12a5648ba4"}
Jul 07 03:17:27.028164 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.028139    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:17:27.105862 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.105834    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedf6d41-af4e-471e-a60e-a6662648cd98-tls-certs\") pod \"dedf6d41-af4e-471e-a60e-a6662648cd98\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") "
Jul 07 03:17:27.105997 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.105880    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dedf6d41-af4e-471e-a60e-a6662648cd98-tokenizer-uds\") pod \"dedf6d41-af4e-471e-a60e-a6662648cd98\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") "
Jul 07 03:17:27.105997 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.105948    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh76l\" (UniqueName: \"kubernetes.io/projected/dedf6d41-af4e-471e-a60e-a6662648cd98-kube-api-access-jh76l\") pod \"dedf6d41-af4e-471e-a60e-a6662648cd98\" (UID: \"dedf6d41-af4e-471e-a60e-a6662648cd98\") "
Jul 07 03:17:27.106156 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.106137    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dedf6d41-af4e-471e-a60e-a6662648cd98-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "dedf6d41-af4e-471e-a60e-a6662648cd98" (UID: "dedf6d41-af4e-471e-a60e-a6662648cd98"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:17:27.106320 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.106305    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dedf6d41-af4e-471e-a60e-a6662648cd98-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:17:27.107948 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.107917    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dedf6d41-af4e-471e-a60e-a6662648cd98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dedf6d41-af4e-471e-a60e-a6662648cd98" (UID: "dedf6d41-af4e-471e-a60e-a6662648cd98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:17:27.108052 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.108011    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dedf6d41-af4e-471e-a60e-a6662648cd98-kube-api-access-jh76l" (OuterVolumeSpecName: "kube-api-access-jh76l") pod "dedf6d41-af4e-471e-a60e-a6662648cd98" (UID: "dedf6d41-af4e-471e-a60e-a6662648cd98"). InnerVolumeSpecName "kube-api-access-jh76l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:17:27.207754 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.207731    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dedf6d41-af4e-471e-a60e-a6662648cd98-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:17:27.207754 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.207751    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh76l\" (UniqueName: \"kubernetes.io/projected/dedf6d41-af4e-471e-a60e-a6662648cd98-kube-api-access-jh76l\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:17:27.928500 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.928472    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"
Jul 07 03:17:27.928905 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.928505    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb" event={"ID":"dedf6d41-af4e-471e-a60e-a6662648cd98","Type":"ContainerDied","Data":"71fb8c9a776d916744d380099bda9bb352db1fc781d0af48701eb9838b174540"}
Jul 07 03:17:27.928905 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.928562    2570 scope.go:117] "RemoveContainer" containerID="c6f3b55038b65962ffb423e08dbacdaacc74125341bdf756b2816c12a5648ba4"
Jul 07 03:17:27.952581 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.952562    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"]
Jul 07 03:17:27.955676 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:27.955655    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-68f87jplvb"]
Jul 07 03:17:29.093152 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:29.093121    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dedf6d41-af4e-471e-a60e-a6662648cd98" path="/var/lib/kubelet/pods/dedf6d41-af4e-471e-a60e-a6662648cd98/volumes"
Jul 07 03:17:49.872256 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.872224    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"]
Jul 07 03:17:49.872672 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.872617    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dedf6d41-af4e-471e-a60e-a6662648cd98" containerName="main"
Jul 07 03:17:49.872672 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.872628    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dedf6d41-af4e-471e-a60e-a6662648cd98" containerName="main"
Jul 07 03:17:49.872760 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.872707    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dedf6d41-af4e-471e-a60e-a6662648cd98" containerName="main"
Jul 07 03:17:49.877072 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.877054    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:49.880314 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.880293    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 07 03:17:49.880434 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.880319    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-x6jt8\""
Jul 07 03:17:49.880434 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.880295    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 03:17:49.880434 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.880292    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 03:17:49.880434 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.880300    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 03:17:49.886736 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.886716    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"]
Jul 07 03:17:49.889460 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.889439    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m76dw\" (UniqueName: \"kubernetes.io/projected/4b37dffe-3597-462f-8d0b-985a267f4987-kube-api-access-m76dw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:49.889573 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.889475    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b37dffe-3597-462f-8d0b-985a267f4987-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:49.889573 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.889522    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b37dffe-3597-462f-8d0b-985a267f4987-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:49.990208 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.990181    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b37dffe-3597-462f-8d0b-985a267f4987-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:49.990298 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.990259    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m76dw\" (UniqueName: \"kubernetes.io/projected/4b37dffe-3597-462f-8d0b-985a267f4987-kube-api-access-m76dw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:49.990298 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.990283    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b37dffe-3597-462f-8d0b-985a267f4987-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:49.990706 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.990681    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b37dffe-3597-462f-8d0b-985a267f4987-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:49.992765 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.992749    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b37dffe-3597-462f-8d0b-985a267f4987-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:49.998411 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:49.998395    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m76dw\" (UniqueName: \"kubernetes.io/projected/4b37dffe-3597-462f-8d0b-985a267f4987-kube-api-access-m76dw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:50.187079 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:50.186999    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:50.342480 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:50.342458    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"]
Jul 07 03:17:50.345084 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:17:50.345058    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b37dffe_3597_462f_8d0b_985a267f4987.slice/crio-0c1823fc475a30214ca24ae41b1440b3b11f044a9e05ba2ad46446a5eb141fed WatchSource:0}: Error finding container 0c1823fc475a30214ca24ae41b1440b3b11f044a9e05ba2ad46446a5eb141fed: Status 404 returned error can't find the container with id 0c1823fc475a30214ca24ae41b1440b3b11f044a9e05ba2ad46446a5eb141fed
Jul 07 03:17:51.048353 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:51.048317    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r" event={"ID":"4b37dffe-3597-462f-8d0b-985a267f4987","Type":"ContainerStarted","Data":"ea0503ee6d5fb79d658f00837c7365df26b4ba2804b8cccd10ee30016448dcba"}
Jul 07 03:17:51.048762 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:51.048392    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r" event={"ID":"4b37dffe-3597-462f-8d0b-985a267f4987","Type":"ContainerStarted","Data":"0c1823fc475a30214ca24ae41b1440b3b11f044a9e05ba2ad46446a5eb141fed"}
Jul 07 03:17:51.048762 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:51.048466    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:17:51.064684 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:17:51.064634    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r" podStartSLOduration=2.064620304 podStartE2EDuration="2.064620304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:17:51.063474269 +0000 UTC m=+3606.471063961" watchObservedRunningTime="2026-07-07 03:17:51.064620304 +0000 UTC m=+3606.472209990"
Jul 07 03:18:22.055512 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:22.055475    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:18:24.004011 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.003981    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"]
Jul 07 03:18:24.013219 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:18:24.013188    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:18:24.013219 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:18:24.013219 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:18:24.013219 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="ea0503ee6d5fb79d658f00837c7365df26b4ba2804b8cccd10ee30016448dcba" cmd=["/bin/sleep","15"]
Jul 07 03:18:24.013501 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.013235    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r" message="" err=<
Jul 07 03:18:24.013501 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:18:24.013501 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:18:24.013501 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:18:24.013501 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:18:24.013247    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:18:24.013501 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:18:24.013501 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:18:24.013501 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r" podUID="4b37dffe-3597-462f-8d0b-985a267f4987" containerName="main" containerID="cri-o://ea0503ee6d5fb79d658f00837c7365df26b4ba2804b8cccd10ee30016448dcba"
Jul 07 03:18:24.013501 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.013290    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r" podUID="4b37dffe-3597-462f-8d0b-985a267f4987" containerName="main" containerID="cri-o://ea0503ee6d5fb79d658f00837c7365df26b4ba2804b8cccd10ee30016448dcba" gracePeriod=60
Jul 07 03:18:24.202524 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.202482    2570 generic.go:356] "Generic (PLEG): container finished" podID="4b37dffe-3597-462f-8d0b-985a267f4987" containerID="ea0503ee6d5fb79d658f00837c7365df26b4ba2804b8cccd10ee30016448dcba" exitCode=0
Jul 07 03:18:24.202645 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.202534    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r" event={"ID":"4b37dffe-3597-462f-8d0b-985a267f4987","Type":"ContainerDied","Data":"ea0503ee6d5fb79d658f00837c7365df26b4ba2804b8cccd10ee30016448dcba"}
Jul 07 03:18:24.268618 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.268570    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:18:24.370860 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.370835    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b37dffe-3597-462f-8d0b-985a267f4987-tokenizer-uds\") pod \"4b37dffe-3597-462f-8d0b-985a267f4987\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") "
Jul 07 03:18:24.370983 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.370893    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b37dffe-3597-462f-8d0b-985a267f4987-tls-certs\") pod \"4b37dffe-3597-462f-8d0b-985a267f4987\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") "
Jul 07 03:18:24.370983 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.370952    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m76dw\" (UniqueName: \"kubernetes.io/projected/4b37dffe-3597-462f-8d0b-985a267f4987-kube-api-access-m76dw\") pod \"4b37dffe-3597-462f-8d0b-985a267f4987\" (UID: \"4b37dffe-3597-462f-8d0b-985a267f4987\") "
Jul 07 03:18:24.371092 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.371069    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b37dffe-3597-462f-8d0b-985a267f4987-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4b37dffe-3597-462f-8d0b-985a267f4987" (UID: "4b37dffe-3597-462f-8d0b-985a267f4987"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:18:24.371173 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.371159    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4b37dffe-3597-462f-8d0b-985a267f4987-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:18:24.373054 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.373025    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b37dffe-3597-462f-8d0b-985a267f4987-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b37dffe-3597-462f-8d0b-985a267f4987" (UID: "4b37dffe-3597-462f-8d0b-985a267f4987"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:18:24.373162 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.373110    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b37dffe-3597-462f-8d0b-985a267f4987-kube-api-access-m76dw" (OuterVolumeSpecName: "kube-api-access-m76dw") pod "4b37dffe-3597-462f-8d0b-985a267f4987" (UID: "4b37dffe-3597-462f-8d0b-985a267f4987"). InnerVolumeSpecName "kube-api-access-m76dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:18:24.471981 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.471958    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m76dw\" (UniqueName: \"kubernetes.io/projected/4b37dffe-3597-462f-8d0b-985a267f4987-kube-api-access-m76dw\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:18:24.471981 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:24.471978    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b37dffe-3597-462f-8d0b-985a267f4987-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:18:25.209038 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:25.209016    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"
Jul 07 03:18:25.209466 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:25.209059    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r" event={"ID":"4b37dffe-3597-462f-8d0b-985a267f4987","Type":"ContainerDied","Data":"0c1823fc475a30214ca24ae41b1440b3b11f044a9e05ba2ad46446a5eb141fed"}
Jul 07 03:18:25.209466 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:25.209117    2570 scope.go:117] "RemoveContainer" containerID="ea0503ee6d5fb79d658f00837c7365df26b4ba2804b8cccd10ee30016448dcba"
Jul 07 03:18:25.225498 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:25.225470    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"]
Jul 07 03:18:25.227470 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:25.227443    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d5wm7r"]
Jul 07 03:18:27.087067 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:27.087030    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b37dffe-3597-462f-8d0b-985a267f4987" path="/var/lib/kubelet/pods/4b37dffe-3597-462f-8d0b-985a267f4987/volumes"
Jul 07 03:18:58.294604 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.294571    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"]
Jul 07 03:18:58.295173 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.295133    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b37dffe-3597-462f-8d0b-985a267f4987" containerName="main"
Jul 07 03:18:58.295173 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.295153    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b37dffe-3597-462f-8d0b-985a267f4987" containerName="main"
Jul 07 03:18:58.295295 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.295256    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b37dffe-3597-462f-8d0b-985a267f4987" containerName="main"
Jul 07 03:18:58.299545 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.299522    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.302806 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.302782    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 03:18:58.302969 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.302945    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 03:18:58.303736 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.303713    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 03:18:58.303906 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.303883    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 07 03:18:58.303994 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.303937    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-72r6x\""
Jul 07 03:18:58.304170 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.304154    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 03:18:58.311337 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.311315    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"]
Jul 07 03:18:58.324897 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.324872    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1903cbb5-b990-46c1-9503-14df1163da92-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.325006 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.324933    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.325006 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.324979    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.325124 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.325055    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmtx\" (UniqueName: \"kubernetes.io/projected/1903cbb5-b990-46c1-9503-14df1163da92-kube-api-access-vvmtx\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.426050 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.426027    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.426150 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.426063    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.426150 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.426099    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmtx\" (UniqueName: \"kubernetes.io/projected/1903cbb5-b990-46c1-9503-14df1163da92-kube-api-access-vvmtx\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.426150 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.426145    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1903cbb5-b990-46c1-9503-14df1163da92-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.426408 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.426391    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.426462 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.426438    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.428795 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.428774    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1903cbb5-b990-46c1-9503-14df1163da92-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.434532 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.434513    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmtx\" (UniqueName: \"kubernetes.io/projected/1903cbb5-b990-46c1-9503-14df1163da92-kube-api-access-vvmtx\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.610222 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.610145    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:18:58.745189 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:58.745168    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"]
Jul 07 03:18:58.747743 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:18:58.747714    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1903cbb5_b990_46c1_9503_14df1163da92.slice/crio-bcc3be34ecb5808810cbb3f451f2240f101f16a4f7e490accf3f2505808d5f3d WatchSource:0}: Error finding container bcc3be34ecb5808810cbb3f451f2240f101f16a4f7e490accf3f2505808d5f3d: Status 404 returned error can't find the container with id bcc3be34ecb5808810cbb3f451f2240f101f16a4f7e490accf3f2505808d5f3d
Jul 07 03:18:59.372216 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:59.372185    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r" event={"ID":"1903cbb5-b990-46c1-9503-14df1163da92","Type":"ContainerStarted","Data":"ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf"}
Jul 07 03:18:59.372606 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:18:59.372236    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r" event={"ID":"1903cbb5-b990-46c1-9503-14df1163da92","Type":"ContainerStarted","Data":"bcc3be34ecb5808810cbb3f451f2240f101f16a4f7e490accf3f2505808d5f3d"}
Jul 07 03:19:00.385889 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:00.385857    2570 generic.go:356] "Generic (PLEG): container finished" podID="1903cbb5-b990-46c1-9503-14df1163da92" containerID="ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf" exitCode=0
Jul 07 03:19:00.386332 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:00.385909    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r" event={"ID":"1903cbb5-b990-46c1-9503-14df1163da92","Type":"ContainerDied","Data":"ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf"}
Jul 07 03:19:01.394864 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:01.394825    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r" event={"ID":"1903cbb5-b990-46c1-9503-14df1163da92","Type":"ContainerStarted","Data":"cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608"}
Jul 07 03:19:01.395293 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:01.394969    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:19:01.417673 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:01.417626    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r" podStartSLOduration=2.774291116 podStartE2EDuration="3.417603784s" totalImagesPullingTime="0s" totalInitContainerRuntime="643.312668ms" isStatefulPod=false podCreationTimestamp="2026-07-07 03:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:19:01.416600699 +0000 UTC m=+3676.824190397" watchObservedRunningTime="2026-07-07 03:19:01.417603784 +0000 UTC m=+3676.825193471"
Jul 07 03:19:24.848518 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:24.848484    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"]
Jul 07 03:19:24.851984 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:24.851965    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:24.854662 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:24.854643    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-epp-sa-dockercfg-9zk2l\""
Jul 07 03:19:24.855462 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:24.855441    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 07 03:19:24.861871 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:24.861852    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"]
Jul 07 03:19:24.925205 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:24.925174    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wdz9\" (UniqueName: \"kubernetes.io/projected/3a5a3e86-155f-4ef6-be93-c44f81a2b708-kube-api-access-8wdz9\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:24.925330 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:24.925226    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:24.925330 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:24.925252    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:24.925485 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:24.925332    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:24.925485 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:24.925419    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.026175 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.026145    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.026277 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.026179    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.026277 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.026202    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.026277 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.026233    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.026475 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.026288    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wdz9\" (UniqueName: \"kubernetes.io/projected/3a5a3e86-155f-4ef6-be93-c44f81a2b708-kube-api-access-8wdz9\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.026657 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.026634    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-uds\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.026657 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.026649    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.026765 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.026686    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.028856 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.028837    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.036848 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.036827    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wdz9\" (UniqueName: \"kubernetes.io/projected/3a5a3e86-155f-4ef6-be93-c44f81a2b708-kube-api-access-8wdz9\") pod \"scheduler-custom-template-test-kserve-router-scheduler-5676f2xn\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.162280 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.162191    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.300182 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.300155    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"]
Jul 07 03:19:25.302504 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:19:25.302463    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5a3e86_155f_4ef6_be93_c44f81a2b708.slice/crio-3acc940d392a5b14eb4ec852d76d31164612fbe86d6f9a02d4d139fad4ba9e07 WatchSource:0}: Error finding container 3acc940d392a5b14eb4ec852d76d31164612fbe86d6f9a02d4d139fad4ba9e07: Status 404 returned error can't find the container with id 3acc940d392a5b14eb4ec852d76d31164612fbe86d6f9a02d4d139fad4ba9e07
Jul 07 03:19:25.515154 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.515066    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn" event={"ID":"3a5a3e86-155f-4ef6-be93-c44f81a2b708","Type":"ContainerStarted","Data":"9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd"}
Jul 07 03:19:25.515154 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.515119    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn" event={"ID":"3a5a3e86-155f-4ef6-be93-c44f81a2b708","Type":"ContainerStarted","Data":"3acc940d392a5b14eb4ec852d76d31164612fbe86d6f9a02d4d139fad4ba9e07"}
Jul 07 03:19:25.515412 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.515162    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:25.532847 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:25.532780    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn" podStartSLOduration=1.532767676 podStartE2EDuration="1.532767676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:19:25.531087403 +0000 UTC m=+3700.938677090" watchObservedRunningTime="2026-07-07 03:19:25.532767676 +0000 UTC m=+3700.940357382"
Jul 07 03:19:32.402327 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:32.402299    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:19:56.521956 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:56.521923    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:57.622036 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.621997    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"]
Jul 07 03:19:57.633240 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:19:57.633204    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:19:57.633240 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:19:57.633240 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:19:57.633240 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd" cmd=["/bin/sleep","15"]
Jul 07 03:19:57.633620 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.633255    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn" message="" err=<
Jul 07 03:19:57.633620 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:19:57.633620 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:19:57.633620 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:19:57.633620 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:19:57.633267    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:19:57.633620 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:19:57.633620 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:19:57.633620 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn" podUID="3a5a3e86-155f-4ef6-be93-c44f81a2b708" containerName="main" containerID="cri-o://9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd"
Jul 07 03:19:57.633620 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.633331    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn" podUID="3a5a3e86-155f-4ef6-be93-c44f81a2b708" containerName="main" containerID="cri-o://9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd" gracePeriod=60
Jul 07 03:19:57.887634 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.887612    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:57.991337 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.991312    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wdz9\" (UniqueName: \"kubernetes.io/projected/3a5a3e86-155f-4ef6-be93-c44f81a2b708-kube-api-access-8wdz9\") pod \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") "
Jul 07 03:19:57.991492 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.991403    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-cachi2-cache\") pod \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") "
Jul 07 03:19:57.991492 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.991436    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-uds\") pod \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") "
Jul 07 03:19:57.991492 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.991482    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tls-certs\") pod \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") "
Jul 07 03:19:57.991683 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.991511    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-cache\") pod \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\" (UID: \"3a5a3e86-155f-4ef6-be93-c44f81a2b708\") "
Jul 07 03:19:57.991683 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.991659    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "3a5a3e86-155f-4ef6-be93-c44f81a2b708" (UID: "3a5a3e86-155f-4ef6-be93-c44f81a2b708"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:19:57.991781 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.991699    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3a5a3e86-155f-4ef6-be93-c44f81a2b708" (UID: "3a5a3e86-155f-4ef6-be93-c44f81a2b708"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:19:57.991781 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.991753    2570 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-cachi2-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:19:57.991875 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.991857    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "3a5a3e86-155f-4ef6-be93-c44f81a2b708" (UID: "3a5a3e86-155f-4ef6-be93-c44f81a2b708"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:19:57.993753 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.993722    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5a3e86-155f-4ef6-be93-c44f81a2b708-kube-api-access-8wdz9" (OuterVolumeSpecName: "kube-api-access-8wdz9") pod "3a5a3e86-155f-4ef6-be93-c44f81a2b708" (UID: "3a5a3e86-155f-4ef6-be93-c44f81a2b708"). InnerVolumeSpecName "kube-api-access-8wdz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:19:57.993863 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:57.993779    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a5a3e86-155f-4ef6-be93-c44f81a2b708" (UID: "3a5a3e86-155f-4ef6-be93-c44f81a2b708"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:19:58.092466 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.092438    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:19:58.092466 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.092459    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:19:58.092466 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.092470    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5a3e86-155f-4ef6-be93-c44f81a2b708-tokenizer-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:19:58.092682 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.092478    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wdz9\" (UniqueName: \"kubernetes.io/projected/3a5a3e86-155f-4ef6-be93-c44f81a2b708-kube-api-access-8wdz9\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:19:58.683182 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.683144    2570 generic.go:356] "Generic (PLEG): container finished" podID="3a5a3e86-155f-4ef6-be93-c44f81a2b708" containerID="9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd" exitCode=0
Jul 07 03:19:58.683662 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.683193    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"
Jul 07 03:19:58.683662 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.683203    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn" event={"ID":"3a5a3e86-155f-4ef6-be93-c44f81a2b708","Type":"ContainerDied","Data":"9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd"}
Jul 07 03:19:58.683662 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.683253    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn" event={"ID":"3a5a3e86-155f-4ef6-be93-c44f81a2b708","Type":"ContainerDied","Data":"3acc940d392a5b14eb4ec852d76d31164612fbe86d6f9a02d4d139fad4ba9e07"}
Jul 07 03:19:58.683662 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.683280    2570 scope.go:117] "RemoveContainer" containerID="9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd"
Jul 07 03:19:58.693328 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.693309    2570 scope.go:117] "RemoveContainer" containerID="9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd"
Jul 07 03:19:58.693636 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:19:58.693606    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd\": container with ID starting with 9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd not found: ID does not exist" containerID="9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd"
Jul 07 03:19:58.693704 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.693645    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd"} err="failed to get container status \"9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd\": rpc error: code = NotFound desc = could not find container \"9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd\": container with ID starting with 9e8fa4b6477f3315645ee29740397177589fd4c0a4f6f498ace9c664348389fd not found: ID does not exist"
Jul 07 03:19:58.704978 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.704955    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"]
Jul 07 03:19:58.708642 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:58.708618    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-router-scheduler-5676f2xn"]
Jul 07 03:19:59.086940 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:19:59.086909    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a5a3e86-155f-4ef6-be93-c44f81a2b708" path="/var/lib/kubelet/pods/3a5a3e86-155f-4ef6-be93-c44f81a2b708/volumes"
Jul 07 03:20:24.464735 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.464701    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"]
Jul 07 03:20:24.465323 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.465136    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5a3e86-155f-4ef6-be93-c44f81a2b708" containerName="main"
Jul 07 03:20:24.465323 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.465149    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5a3e86-155f-4ef6-be93-c44f81a2b708" containerName="main"
Jul 07 03:20:24.465323 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.465221    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5a3e86-155f-4ef6-be93-c44f81a2b708" containerName="main"
Jul 07 03:20:24.468632 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.468607    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.471261 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.471171    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-9x256\""
Jul 07 03:20:24.471261 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.471203    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 07 03:20:24.478883 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.478859    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"]
Jul 07 03:20:24.508489 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.508465    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.508612 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.508495    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.508612 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.508527    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.508612 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.508553    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.508612 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.508577    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp7wt\" (UniqueName: \"kubernetes.io/projected/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-kube-api-access-wp7wt\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.508612 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.508599    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.609570 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.609548    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.609724 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.609576    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.609724 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.609599    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp7wt\" (UniqueName: \"kubernetes.io/projected/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-kube-api-access-wp7wt\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.609724 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.609631    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.609724 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.609713    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.609938 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.609736    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.610002 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.609948    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-home\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.610059 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.610029    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.610122 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.610102    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.612122 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.612098    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.612328 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.612310    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.617233 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.617214    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp7wt\" (UniqueName: \"kubernetes.io/projected/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-kube-api-access-wp7wt\") pod \"scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.780633 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.780601    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:24.916982 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.916958    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"]
Jul 07 03:20:24.919299 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:20:24.919263    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92ecb24_ab1b_41ef_a9dd_fe357c964fb1.slice/crio-b387196f122e2ba16f798e79636f7fef84fd6b6d92668767a850c0703f863211 WatchSource:0}: Error finding container b387196f122e2ba16f798e79636f7fef84fd6b6d92668767a850c0703f863211: Status 404 returned error can't find the container with id b387196f122e2ba16f798e79636f7fef84fd6b6d92668767a850c0703f863211
Jul 07 03:20:24.921601 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:24.921580    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 03:20:25.821529 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:25.821493    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw" event={"ID":"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1","Type":"ContainerStarted","Data":"234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e"}
Jul 07 03:20:25.821893 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:25.821561    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:25.821893 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:25.821588    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw" event={"ID":"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1","Type":"ContainerStarted","Data":"b387196f122e2ba16f798e79636f7fef84fd6b6d92668767a850c0703f863211"}
Jul 07 03:20:26.828697 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:26.828660    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw" event={"ID":"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1","Type":"ContainerStarted","Data":"be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf"}
Jul 07 03:20:26.847993 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:26.847933    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw" podStartSLOduration=2.128645387 podStartE2EDuration="2.847916347s" totalImagesPullingTime="719.27096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:20:26.846021957 +0000 UTC m=+3762.253611644" watchObservedRunningTime="2026-07-07 03:20:26.847916347 +0000 UTC m=+3762.255506033"
Jul 07 03:20:27.780841 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:27.780809    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:34.781549 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:34.781516    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:34.793846 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:34.793824    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:34.878069 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:34.878047    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:35.884412 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:35.884388    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:20:58.873361 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:20:58.873326    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"]
Jul 07 03:21:13.894039 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:13.894000    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw" podUID="f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" containerName="main" containerID="cri-o://be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf" gracePeriod=45
Jul 07 03:21:14.005469 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.005415    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw" podUID="f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" containerName="llm-d-routing-sidecar" containerID="cri-o://234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e" gracePeriod=45
Jul 07 03:21:14.073213 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.073190    2570 generic.go:356] "Generic (PLEG): container finished" podID="f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" containerID="be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf" exitCode=0
Jul 07 03:21:14.073301 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.073270    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw" event={"ID":"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1","Type":"ContainerDied","Data":"be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf"}
Jul 07 03:21:14.260419 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.260389    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:21:14.421128 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.421099    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-dshm\") pod \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") "
Jul 07 03:21:14.421128 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.421129    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-home\") pod \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") "
Jul 07 03:21:14.421374 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.421172    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp7wt\" (UniqueName: \"kubernetes.io/projected/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-kube-api-access-wp7wt\") pod \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") "
Jul 07 03:21:14.421374 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.421223    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-model-cache\") pod \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") "
Jul 07 03:21:14.421374 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.421254    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tls-certs\") pod \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") "
Jul 07 03:21:14.421374 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.421268    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tmp-dir\") pod \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\" (UID: \"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1\") "
Jul 07 03:21:14.421589 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.421439    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-home" (OuterVolumeSpecName: "home") pod "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" (UID: "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:21:14.421589 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.421535    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-model-cache" (OuterVolumeSpecName: "model-cache") pod "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" (UID: "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:21:14.421708 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.421641    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" (UID: "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:21:14.421708 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.421660    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:21:14.421708 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.421678    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-home\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:21:14.423525 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.423499    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-dshm" (OuterVolumeSpecName: "dshm") pod "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" (UID: "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:21:14.423525 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.423520    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-kube-api-access-wp7wt" (OuterVolumeSpecName: "kube-api-access-wp7wt") pod "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" (UID: "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1"). InnerVolumeSpecName "kube-api-access-wp7wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:21:14.423672 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.423530    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" (UID: "f92ecb24-ab1b-41ef-a9dd-fe357c964fb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:21:14.522500 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.522478    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:21:14.522500 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.522497    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-tmp-dir\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:21:14.522629 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.522505    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-dshm\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:21:14.522629 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:14.522514    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp7wt\" (UniqueName: \"kubernetes.io/projected/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1-kube-api-access-wp7wt\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:21:15.080661 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.080631    2570 generic.go:356] "Generic (PLEG): container finished" podID="f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" containerID="234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e" exitCode=0
Jul 07 03:21:15.082940 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.082918    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"
Jul 07 03:21:15.086783 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.086762    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw" event={"ID":"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1","Type":"ContainerDied","Data":"234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e"}
Jul 07 03:21:15.086885 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.086801    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw" event={"ID":"f92ecb24-ab1b-41ef-a9dd-fe357c964fb1","Type":"ContainerDied","Data":"b387196f122e2ba16f798e79636f7fef84fd6b6d92668767a850c0703f863211"}
Jul 07 03:21:15.086885 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.086830    2570 scope.go:117] "RemoveContainer" containerID="be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf"
Jul 07 03:21:15.096444 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.096429    2570 scope.go:117] "RemoveContainer" containerID="234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e"
Jul 07 03:21:15.104266 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.104245    2570 scope.go:117] "RemoveContainer" containerID="be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf"
Jul 07 03:21:15.104539 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:21:15.104510    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf\": container with ID starting with be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf not found: ID does not exist" containerID="be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf"
Jul 07 03:21:15.104604 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.104547    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf"} err="failed to get container status \"be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf\": rpc error: code = NotFound desc = could not find container \"be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf\": container with ID starting with be51aa171d1359cc72735b3cebdd69796addbdec52b3f46329799ef0197856bf not found: ID does not exist"
Jul 07 03:21:15.104604 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.104562    2570 scope.go:117] "RemoveContainer" containerID="234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e"
Jul 07 03:21:15.104812 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:21:15.104794    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e\": container with ID starting with 234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e not found: ID does not exist" containerID="234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e"
Jul 07 03:21:15.104854 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.104819    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e"} err="failed to get container status \"234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e\": rpc error: code = NotFound desc = could not find container \"234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e\": container with ID starting with 234a3b2d57d13a9a6e6d712ed3331f692580a47316ef43b86cd9a274eec3ed5e not found: ID does not exist"
Jul 07 03:21:15.108428 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.108409    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"]
Jul 07 03:21:15.112542 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:15.112521    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-5c87756974-tnfnw"]
Jul 07 03:21:17.093818 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:17.093783    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" path="/var/lib/kubelet/pods/f92ecb24-ab1b-41ef-a9dd-fe357c964fb1/volumes"
Jul 07 03:21:30.375891 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.375856    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"]
Jul 07 03:21:30.376600 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.376258    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" containerName="main"
Jul 07 03:21:30.376600 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.376271    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" containerName="main"
Jul 07 03:21:30.376600 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.376294    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" containerName="llm-d-routing-sidecar"
Jul 07 03:21:30.376600 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.376300    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" containerName="llm-d-routing-sidecar"
Jul 07 03:21:30.376600 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.376394    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" containerName="llm-d-routing-sidecar"
Jul 07 03:21:30.376600 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.376409    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f92ecb24-ab1b-41ef-a9dd-fe357c964fb1" containerName="main"
Jul 07 03:21:30.379524 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.379507    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.382474 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.382449    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 07 03:21:30.383290 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.383273    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-7sccv\""
Jul 07 03:21:30.404118 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.404094    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"]
Jul 07 03:21:30.448738 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.448709    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98q2d\" (UniqueName: \"kubernetes.io/projected/a9eec126-6762-454d-9601-b5ea937c64a0-kube-api-access-98q2d\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.448870 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.448773    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.448870 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.448803    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9eec126-6762-454d-9601-b5ea937c64a0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.448870 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.448835    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.448984 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.448910    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.448984 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.448954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.549635 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.549596    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.549755 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.549638    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9eec126-6762-454d-9601-b5ea937c64a0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.549755 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.549662    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.549755 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.549683    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.549755 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.549701    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.549755 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.549744    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98q2d\" (UniqueName: \"kubernetes.io/projected/a9eec126-6762-454d-9601-b5ea937c64a0-kube-api-access-98q2d\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.550109 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.550087    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.550166 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.550130    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.550166 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.550155    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.551919 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.551901    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.552182 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.552167    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9eec126-6762-454d-9601-b5ea937c64a0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.561033 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.561013    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98q2d\" (UniqueName: \"kubernetes.io/projected/a9eec126-6762-454d-9601-b5ea937c64a0-kube-api-access-98q2d\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.689667 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.689611    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:30.828064 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:30.828029    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"]
Jul 07 03:21:30.831207 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:21:30.831179    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9eec126_6762_454d_9601_b5ea937c64a0.slice/crio-3acdab1375e8955ca0a9ad444c5cc02d26127462b22c36860f2a0657ca9afe73 WatchSource:0}: Error finding container 3acdab1375e8955ca0a9ad444c5cc02d26127462b22c36860f2a0657ca9afe73: Status 404 returned error can't find the container with id 3acdab1375e8955ca0a9ad444c5cc02d26127462b22c36860f2a0657ca9afe73
Jul 07 03:21:31.168447 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:31.168413    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996" event={"ID":"a9eec126-6762-454d-9601-b5ea937c64a0","Type":"ContainerStarted","Data":"70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93"}
Jul 07 03:21:31.168648 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:31.168500    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996" event={"ID":"a9eec126-6762-454d-9601-b5ea937c64a0","Type":"ContainerStarted","Data":"3acdab1375e8955ca0a9ad444c5cc02d26127462b22c36860f2a0657ca9afe73"}
Jul 07 03:21:31.168648 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:31.168533    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:32.176058 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:32.176019    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996" event={"ID":"a9eec126-6762-454d-9601-b5ea937c64a0","Type":"ContainerStarted","Data":"549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1"}
Jul 07 03:21:32.196064 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:32.196012    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996" podStartSLOduration=2.195992973 podStartE2EDuration="2.195992973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:21:32.192573826 +0000 UTC m=+3827.600163513" watchObservedRunningTime="2026-07-07 03:21:32.195992973 +0000 UTC m=+3827.603582661"
Jul 07 03:21:32.689830 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:32.689800    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:33.565271 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.565240    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"]
Jul 07 03:21:33.575141 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:21:33.575102    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:21:33.575141 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:21:33.575141 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:21:33.575141 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608" cmd=["/bin/sleep","15"]
Jul 07 03:21:33.575633 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.575155    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r" message="" err=<
Jul 07 03:21:33.575633 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:21:33.575633 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:21:33.575633 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:21:33.575633 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:21:33.575173    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:21:33.575633 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:21:33.575633 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:21:33.575633 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r" podUID="1903cbb5-b990-46c1-9503-14df1163da92" containerName="main" containerID="cri-o://cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608"
Jul 07 03:21:33.575633 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.575216    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r" podUID="1903cbb5-b990-46c1-9503-14df1163da92" containerName="main" containerID="cri-o://cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608" gracePeriod=60
Jul 07 03:21:33.825546 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.825494    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:21:33.878219 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.878199    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-kserve-provision-location\") pod \"1903cbb5-b990-46c1-9503-14df1163da92\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") "
Jul 07 03:21:33.878339 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.878262    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-tokenizer-uds\") pod \"1903cbb5-b990-46c1-9503-14df1163da92\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") "
Jul 07 03:21:33.878339 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.878307    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvmtx\" (UniqueName: \"kubernetes.io/projected/1903cbb5-b990-46c1-9503-14df1163da92-kube-api-access-vvmtx\") pod \"1903cbb5-b990-46c1-9503-14df1163da92\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") "
Jul 07 03:21:33.878463 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.878426    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1903cbb5-b990-46c1-9503-14df1163da92-tls-certs\") pod \"1903cbb5-b990-46c1-9503-14df1163da92\" (UID: \"1903cbb5-b990-46c1-9503-14df1163da92\") "
Jul 07 03:21:33.878552 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.878526    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1903cbb5-b990-46c1-9503-14df1163da92" (UID: "1903cbb5-b990-46c1-9503-14df1163da92"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:21:33.878703 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.878686    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:21:33.878911 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.878891    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1903cbb5-b990-46c1-9503-14df1163da92" (UID: "1903cbb5-b990-46c1-9503-14df1163da92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:21:33.880555 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.880536    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1903cbb5-b990-46c1-9503-14df1163da92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1903cbb5-b990-46c1-9503-14df1163da92" (UID: "1903cbb5-b990-46c1-9503-14df1163da92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:21:33.880621 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.880564    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1903cbb5-b990-46c1-9503-14df1163da92-kube-api-access-vvmtx" (OuterVolumeSpecName: "kube-api-access-vvmtx") pod "1903cbb5-b990-46c1-9503-14df1163da92" (UID: "1903cbb5-b990-46c1-9503-14df1163da92"). InnerVolumeSpecName "kube-api-access-vvmtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:21:33.979129 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.979109    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmtx\" (UniqueName: \"kubernetes.io/projected/1903cbb5-b990-46c1-9503-14df1163da92-kube-api-access-vvmtx\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:21:33.979129 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.979129    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1903cbb5-b990-46c1-9503-14df1163da92-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:21:33.979245 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:33.979140    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1903cbb5-b990-46c1-9503-14df1163da92-kserve-provision-location\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:21:34.187578 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.187436    2570 generic.go:356] "Generic (PLEG): container finished" podID="1903cbb5-b990-46c1-9503-14df1163da92" containerID="cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608" exitCode=0
Jul 07 03:21:34.187578 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.187483    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"
Jul 07 03:21:34.187578 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.187516    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r" event={"ID":"1903cbb5-b990-46c1-9503-14df1163da92","Type":"ContainerDied","Data":"cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608"}
Jul 07 03:21:34.187578 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.187563    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r" event={"ID":"1903cbb5-b990-46c1-9503-14df1163da92","Type":"ContainerDied","Data":"bcc3be34ecb5808810cbb3f451f2240f101f16a4f7e490accf3f2505808d5f3d"}
Jul 07 03:21:34.209519 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.187584    2570 scope.go:117] "RemoveContainer" containerID="cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608"
Jul 07 03:21:34.209519 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.197251    2570 scope.go:117] "RemoveContainer" containerID="ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf"
Jul 07 03:21:34.209519 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.205036    2570 scope.go:117] "RemoveContainer" containerID="cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608"
Jul 07 03:21:34.209519 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:21:34.205284    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608\": container with ID starting with cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608 not found: ID does not exist" containerID="cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608"
Jul 07 03:21:34.209519 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.205303    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608"} err="failed to get container status \"cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608\": rpc error: code = NotFound desc = could not find container \"cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608\": container with ID starting with cec61523ef8d23effe0b82702149cb5ff6d157ff7028bab92148fd312ca11608 not found: ID does not exist"
Jul 07 03:21:34.209519 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.205318    2570 scope.go:117] "RemoveContainer" containerID="ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf"
Jul 07 03:21:34.209519 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:21:34.205539    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf\": container with ID starting with ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf not found: ID does not exist" containerID="ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf"
Jul 07 03:21:34.209519 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.205560    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf"} err="failed to get container status \"ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf\": rpc error: code = NotFound desc = could not find container \"ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf\": container with ID starting with ab5004f90420895116191755f1bcb198f0776e2a9aee8433634a25176b938edf not found: ID does not exist"
Jul 07 03:21:34.209519 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.208625    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"]
Jul 07 03:21:34.210720 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:34.210701    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche7vr9r"]
Jul 07 03:21:35.088312 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:35.088281    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1903cbb5-b990-46c1-9503-14df1163da92" path="/var/lib/kubelet/pods/1903cbb5-b990-46c1-9503-14df1163da92/volumes"
Jul 07 03:21:40.690293 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:40.690258    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:40.702807 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:40.702783    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:40.706484 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:40.706465    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:21:41.234415 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:21:41.234378    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:22:04.361823 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:04.361791    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"]
Jul 07 03:22:19.382790 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.382723    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996" podUID="a9eec126-6762-454d-9601-b5ea937c64a0" containerName="main" containerID="cri-o://549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1" gracePeriod=45
Jul 07 03:22:19.499989 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.499954    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996" podUID="a9eec126-6762-454d-9601-b5ea937c64a0" containerName="llm-d-routing-sidecar" containerID="cri-o://70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93" gracePeriod=45
Jul 07 03:22:19.755067 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.755044    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:22:19.850398 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.850369    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-tmp-dir\") pod \"a9eec126-6762-454d-9601-b5ea937c64a0\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") "
Jul 07 03:22:19.850522 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.850422    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98q2d\" (UniqueName: \"kubernetes.io/projected/a9eec126-6762-454d-9601-b5ea937c64a0-kube-api-access-98q2d\") pod \"a9eec126-6762-454d-9601-b5ea937c64a0\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") "
Jul 07 03:22:19.850522 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.850450    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-home\") pod \"a9eec126-6762-454d-9601-b5ea937c64a0\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") "
Jul 07 03:22:19.850522 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.850468    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9eec126-6762-454d-9601-b5ea937c64a0-tls-certs\") pod \"a9eec126-6762-454d-9601-b5ea937c64a0\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") "
Jul 07 03:22:19.850522 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.850509    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-model-cache\") pod \"a9eec126-6762-454d-9601-b5ea937c64a0\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") "
Jul 07 03:22:19.850737 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.850529    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-dshm\") pod \"a9eec126-6762-454d-9601-b5ea937c64a0\" (UID: \"a9eec126-6762-454d-9601-b5ea937c64a0\") "
Jul 07 03:22:19.850737 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.850646    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9eec126-6762-454d-9601-b5ea937c64a0" (UID: "a9eec126-6762-454d-9601-b5ea937c64a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:22:19.850737 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.850684    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-home" (OuterVolumeSpecName: "home") pod "a9eec126-6762-454d-9601-b5ea937c64a0" (UID: "a9eec126-6762-454d-9601-b5ea937c64a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:22:19.850878 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.850821    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-home\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:22:19.850878 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.850838    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-tmp-dir\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:22:19.850972 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.850874    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9eec126-6762-454d-9601-b5ea937c64a0" (UID: "a9eec126-6762-454d-9601-b5ea937c64a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:22:19.852743 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.852719    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-dshm" (OuterVolumeSpecName: "dshm") pod "a9eec126-6762-454d-9601-b5ea937c64a0" (UID: "a9eec126-6762-454d-9601-b5ea937c64a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:22:19.852844 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.852736    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9eec126-6762-454d-9601-b5ea937c64a0-kube-api-access-98q2d" (OuterVolumeSpecName: "kube-api-access-98q2d") pod "a9eec126-6762-454d-9601-b5ea937c64a0" (UID: "a9eec126-6762-454d-9601-b5ea937c64a0"). InnerVolumeSpecName "kube-api-access-98q2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:22:19.852844 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.852772    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9eec126-6762-454d-9601-b5ea937c64a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9eec126-6762-454d-9601-b5ea937c64a0" (UID: "a9eec126-6762-454d-9601-b5ea937c64a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:22:19.951275 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.951217    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98q2d\" (UniqueName: \"kubernetes.io/projected/a9eec126-6762-454d-9601-b5ea937c64a0-kube-api-access-98q2d\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:22:19.951275 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.951239    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9eec126-6762-454d-9601-b5ea937c64a0-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:22:19.951275 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.951249    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:22:19.951275 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:19.951256    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9eec126-6762-454d-9601-b5ea937c64a0-dshm\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:22:20.422922 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.422893    2570 generic.go:356] "Generic (PLEG): container finished" podID="a9eec126-6762-454d-9601-b5ea937c64a0" containerID="549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1" exitCode=0
Jul 07 03:22:20.422922 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.422917    2570 generic.go:356] "Generic (PLEG): container finished" podID="a9eec126-6762-454d-9601-b5ea937c64a0" containerID="70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93" exitCode=0
Jul 07 03:22:20.423381 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.422944    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"
Jul 07 03:22:20.423381 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.422964    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996" event={"ID":"a9eec126-6762-454d-9601-b5ea937c64a0","Type":"ContainerDied","Data":"549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1"}
Jul 07 03:22:20.423381 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.423014    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996" event={"ID":"a9eec126-6762-454d-9601-b5ea937c64a0","Type":"ContainerDied","Data":"70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93"}
Jul 07 03:22:20.423381 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.423032    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996" event={"ID":"a9eec126-6762-454d-9601-b5ea937c64a0","Type":"ContainerDied","Data":"3acdab1375e8955ca0a9ad444c5cc02d26127462b22c36860f2a0657ca9afe73"}
Jul 07 03:22:20.423381 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.423052    2570 scope.go:117] "RemoveContainer" containerID="549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1"
Jul 07 03:22:20.432037 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.432022    2570 scope.go:117] "RemoveContainer" containerID="70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93"
Jul 07 03:22:20.440098 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.440081    2570 scope.go:117] "RemoveContainer" containerID="549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1"
Jul 07 03:22:20.440366 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:22:20.440331    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1\": container with ID starting with 549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1 not found: ID does not exist" containerID="549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1"
Jul 07 03:22:20.440420 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.440375    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1"} err="failed to get container status \"549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1\": rpc error: code = NotFound desc = could not find container \"549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1\": container with ID starting with 549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1 not found: ID does not exist"
Jul 07 03:22:20.440420 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.440393    2570 scope.go:117] "RemoveContainer" containerID="70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93"
Jul 07 03:22:20.440636 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:22:20.440616    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93\": container with ID starting with 70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93 not found: ID does not exist" containerID="70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93"
Jul 07 03:22:20.440706 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.440644    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93"} err="failed to get container status \"70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93\": rpc error: code = NotFound desc = could not find container \"70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93\": container with ID starting with 70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93 not found: ID does not exist"
Jul 07 03:22:20.440706 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.440666    2570 scope.go:117] "RemoveContainer" containerID="549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1"
Jul 07 03:22:20.440915 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.440897    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1"} err="failed to get container status \"549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1\": rpc error: code = NotFound desc = could not find container \"549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1\": container with ID starting with 549b5301d686fd1c826f4c2a6b68632570100d602941a1ce52120383787465b1 not found: ID does not exist"
Jul 07 03:22:20.440962 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.440916    2570 scope.go:117] "RemoveContainer" containerID="70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93"
Jul 07 03:22:20.441133 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.441115    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93"} err="failed to get container status \"70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93\": rpc error: code = NotFound desc = could not find container \"70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93\": container with ID starting with 70798b8a2854e3ccf88760a20b485c7c577ed3cd5441c4e5d5e2ebd0fa391a93 not found: ID does not exist"
Jul 07 03:22:20.444108 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.444090    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"]
Jul 07 03:22:20.447239 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:20.447220    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-9k996"]
Jul 07 03:22:21.086419 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:21.086391    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9eec126-6762-454d-9601-b5ea937c64a0" path="/var/lib/kubelet/pods/a9eec126-6762-454d-9601-b5ea937c64a0/volumes"
Jul 07 03:22:53.655370 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655320    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"]
Jul 07 03:22:53.655953 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655776    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1903cbb5-b990-46c1-9503-14df1163da92" containerName="storage-initializer"
Jul 07 03:22:53.655953 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655796    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1903cbb5-b990-46c1-9503-14df1163da92" containerName="storage-initializer"
Jul 07 03:22:53.655953 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655809    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1903cbb5-b990-46c1-9503-14df1163da92" containerName="main"
Jul 07 03:22:53.655953 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655818    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1903cbb5-b990-46c1-9503-14df1163da92" containerName="main"
Jul 07 03:22:53.655953 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655827    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9eec126-6762-454d-9601-b5ea937c64a0" containerName="main"
Jul 07 03:22:53.655953 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655832    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9eec126-6762-454d-9601-b5ea937c64a0" containerName="main"
Jul 07 03:22:53.655953 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655849    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9eec126-6762-454d-9601-b5ea937c64a0" containerName="llm-d-routing-sidecar"
Jul 07 03:22:53.655953 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655856    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9eec126-6762-454d-9601-b5ea937c64a0" containerName="llm-d-routing-sidecar"
Jul 07 03:22:53.655953 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655928    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1903cbb5-b990-46c1-9503-14df1163da92" containerName="main"
Jul 07 03:22:53.655953 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655940    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9eec126-6762-454d-9601-b5ea937c64a0" containerName="llm-d-routing-sidecar"
Jul 07 03:22:53.655953 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.655947    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9eec126-6762-454d-9601-b5ea937c64a0" containerName="main"
Jul 07 03:22:53.660294 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.660276    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:53.662672 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.662650    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 03:22:53.662773 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.662693    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 03:22:53.663533 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.663518    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 03:22:53.663636 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.663517    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-mzcqw\""
Jul 07 03:22:53.663636 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.663600    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 07 03:22:53.668577 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.668560    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"]
Jul 07 03:22:53.712133 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.712109    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:53.712253 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.712156    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n627x\" (UniqueName: \"kubernetes.io/projected/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-kube-api-access-n627x\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:53.712253 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.712219    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:53.813196 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.813164    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:53.813369 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.813206    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n627x\" (UniqueName: \"kubernetes.io/projected/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-kube-api-access-n627x\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:53.813369 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.813228    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:53.813593 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.813574    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:53.815710 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.815678    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:53.821361 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.821321    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n627x\" (UniqueName: \"kubernetes.io/projected/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-kube-api-access-n627x\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:53.971286 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:53.971223    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:54.102645 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:54.102614    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"]
Jul 07 03:22:54.104948 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:22:54.104922    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa90685_b3f1_4117_94f5_d4079e2cfe5b.slice/crio-07ece9b5f0d3a8ef857da65d04ad9c21afb5cf4216ad978ce15c0946491a2b2e WatchSource:0}: Error finding container 07ece9b5f0d3a8ef857da65d04ad9c21afb5cf4216ad978ce15c0946491a2b2e: Status 404 returned error can't find the container with id 07ece9b5f0d3a8ef857da65d04ad9c21afb5cf4216ad978ce15c0946491a2b2e
Jul 07 03:22:54.593131 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:54.593099    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt" event={"ID":"8aa90685-b3f1-4117-94f5-d4079e2cfe5b","Type":"ContainerStarted","Data":"915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8"}
Jul 07 03:22:54.593315 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:54.593143    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt" event={"ID":"8aa90685-b3f1-4117-94f5-d4079e2cfe5b","Type":"ContainerStarted","Data":"07ece9b5f0d3a8ef857da65d04ad9c21afb5cf4216ad978ce15c0946491a2b2e"}
Jul 07 03:22:54.593315 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:54.593240    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:22:54.611294 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:22:54.611241    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt" podStartSLOduration=1.611223324 podStartE2EDuration="1.611223324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:22:54.610886167 +0000 UTC m=+3910.018475855" watchObservedRunningTime="2026-07-07 03:22:54.611223324 +0000 UTC m=+3910.018813012"
Jul 07 03:23:25.600335 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:25.600303    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:23:26.677900 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:26.677873    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"]
Jul 07 03:23:26.687464 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:23:26.687429    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:23:26.687464 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:23:26.687464 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:23:26.687464 ip-10-0-141-229 kubenswrapper[2570]:  > containerID="915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8" cmd=["/bin/sleep","15"]
Jul 07 03:23:26.687725 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:26.687478    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt" message="" err=<
Jul 07 03:23:26.687725 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:23:26.687725 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:23:26.687725 ip-10-0-141-229 kubenswrapper[2570]:  >
Jul 07 03:23:26.687725 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:23:26.687490    2570 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:23:26.687725 ip-10-0-141-229 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:23:26.687725 ip-10-0-141-229 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Jul 07 03:23:26.687725 ip-10-0-141-229 kubenswrapper[2570]:  > pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt" podUID="8aa90685-b3f1-4117-94f5-d4079e2cfe5b" containerName="main" containerID="cri-o://915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8"
Jul 07 03:23:26.687725 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:26.687525    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt" podUID="8aa90685-b3f1-4117-94f5-d4079e2cfe5b" containerName="main" containerID="cri-o://915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8" gracePeriod=60
Jul 07 03:23:26.931997 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:26.931940    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:23:26.983956 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:26.983934    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n627x\" (UniqueName: \"kubernetes.io/projected/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-kube-api-access-n627x\") pod \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") "
Jul 07 03:23:26.984086 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:26.983979    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tokenizer-uds\") pod \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") "
Jul 07 03:23:26.984086 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:26.984045    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tls-certs\") pod \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\" (UID: \"8aa90685-b3f1-4117-94f5-d4079e2cfe5b\") "
Jul 07 03:23:26.984264 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:26.984243    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8aa90685-b3f1-4117-94f5-d4079e2cfe5b" (UID: "8aa90685-b3f1-4117-94f5-d4079e2cfe5b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:23:26.986150 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:26.986117    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8aa90685-b3f1-4117-94f5-d4079e2cfe5b" (UID: "8aa90685-b3f1-4117-94f5-d4079e2cfe5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:23:26.986261 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:26.986172    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-kube-api-access-n627x" (OuterVolumeSpecName: "kube-api-access-n627x") pod "8aa90685-b3f1-4117-94f5-d4079e2cfe5b" (UID: "8aa90685-b3f1-4117-94f5-d4079e2cfe5b"). InnerVolumeSpecName "kube-api-access-n627x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:23:27.084657 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.084632    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tokenizer-uds\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:23:27.084657 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.084652    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:23:27.084789 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.084662    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n627x\" (UniqueName: \"kubernetes.io/projected/8aa90685-b3f1-4117-94f5-d4079e2cfe5b-kube-api-access-n627x\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:23:27.758694 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.758659    2570 generic.go:356] "Generic (PLEG): container finished" podID="8aa90685-b3f1-4117-94f5-d4079e2cfe5b" containerID="915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8" exitCode=0
Jul 07 03:23:27.759164 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.758701    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt" event={"ID":"8aa90685-b3f1-4117-94f5-d4079e2cfe5b","Type":"ContainerDied","Data":"915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8"}
Jul 07 03:23:27.759164 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.758710    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"
Jul 07 03:23:27.759164 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.758761    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt" event={"ID":"8aa90685-b3f1-4117-94f5-d4079e2cfe5b","Type":"ContainerDied","Data":"07ece9b5f0d3a8ef857da65d04ad9c21afb5cf4216ad978ce15c0946491a2b2e"}
Jul 07 03:23:27.759164 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.758781    2570 scope.go:117] "RemoveContainer" containerID="915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8"
Jul 07 03:23:27.767763 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.767743    2570 scope.go:117] "RemoveContainer" containerID="915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8"
Jul 07 03:23:27.767987 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:23:27.767964    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8\": container with ID starting with 915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8 not found: ID does not exist" containerID="915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8"
Jul 07 03:23:27.768088 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.767990    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8"} err="failed to get container status \"915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8\": rpc error: code = NotFound desc = could not find container \"915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8\": container with ID starting with 915d244485c971ff91dcba5e65b0169020aba30cbe5acfe6cbd81be5a54104e8 not found: ID does not exist"
Jul 07 03:23:27.773699 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.773678    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"]
Jul 07 03:23:27.777468 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:27.777450    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8zhmt"]
Jul 07 03:23:29.086967 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:29.086937    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa90685-b3f1-4117-94f5-d4079e2cfe5b" path="/var/lib/kubelet/pods/8aa90685-b3f1-4117-94f5-d4079e2cfe5b/volumes"
Jul 07 03:23:48.164358 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.164321    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9jjq7/must-gather-6k7n4"]
Jul 07 03:23:48.164827 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.164699    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa90685-b3f1-4117-94f5-d4079e2cfe5b" containerName="main"
Jul 07 03:23:48.164827 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.164710    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa90685-b3f1-4117-94f5-d4079e2cfe5b" containerName="main"
Jul 07 03:23:48.164827 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.164770    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa90685-b3f1-4117-94f5-d4079e2cfe5b" containerName="main"
Jul 07 03:23:48.168949 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.168931    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:23:48.171364 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.171326    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9jjq7\"/\"kube-root-ca.crt\""
Jul 07 03:23:48.171458 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.171403    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9jjq7\"/\"openshift-service-ca.crt\""
Jul 07 03:23:48.176368 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.176330    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9jjq7/must-gather-6k7n4"]
Jul 07 03:23:48.247877 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.247849    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5p7\" (UniqueName: \"kubernetes.io/projected/26330820-72c0-4119-8d47-3164fd6ed91f-kube-api-access-gf5p7\") pod \"must-gather-6k7n4\" (UID: \"26330820-72c0-4119-8d47-3164fd6ed91f\") " pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:23:48.247975 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.247897    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/26330820-72c0-4119-8d47-3164fd6ed91f-must-gather-output\") pod \"must-gather-6k7n4\" (UID: \"26330820-72c0-4119-8d47-3164fd6ed91f\") " pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:23:48.348504 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.348480    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/26330820-72c0-4119-8d47-3164fd6ed91f-must-gather-output\") pod \"must-gather-6k7n4\" (UID: \"26330820-72c0-4119-8d47-3164fd6ed91f\") " pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:23:48.348592 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.348539    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5p7\" (UniqueName: \"kubernetes.io/projected/26330820-72c0-4119-8d47-3164fd6ed91f-kube-api-access-gf5p7\") pod \"must-gather-6k7n4\" (UID: \"26330820-72c0-4119-8d47-3164fd6ed91f\") " pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:23:48.348797 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.348775    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/26330820-72c0-4119-8d47-3164fd6ed91f-must-gather-output\") pod \"must-gather-6k7n4\" (UID: \"26330820-72c0-4119-8d47-3164fd6ed91f\") " pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:23:48.355986 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.355961    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5p7\" (UniqueName: \"kubernetes.io/projected/26330820-72c0-4119-8d47-3164fd6ed91f-kube-api-access-gf5p7\") pod \"must-gather-6k7n4\" (UID: \"26330820-72c0-4119-8d47-3164fd6ed91f\") " pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:23:48.478840 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.478782    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:23:48.611321 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.611296    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9jjq7/must-gather-6k7n4"]
Jul 07 03:23:48.613414 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:23:48.613390    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26330820_72c0_4119_8d47_3164fd6ed91f.slice/crio-7a6e070269c2ee0662f91d019fe2813efdbe6f7fde117c116abeb1332b026f08 WatchSource:0}: Error finding container 7a6e070269c2ee0662f91d019fe2813efdbe6f7fde117c116abeb1332b026f08: Status 404 returned error can't find the container with id 7a6e070269c2ee0662f91d019fe2813efdbe6f7fde117c116abeb1332b026f08
Jul 07 03:23:48.862900 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:48.862870    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jjq7/must-gather-6k7n4" event={"ID":"26330820-72c0-4119-8d47-3164fd6ed91f","Type":"ContainerStarted","Data":"7a6e070269c2ee0662f91d019fe2813efdbe6f7fde117c116abeb1332b026f08"}
Jul 07 03:23:55.908034 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:55.907997    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jjq7/must-gather-6k7n4" event={"ID":"26330820-72c0-4119-8d47-3164fd6ed91f","Type":"ContainerStarted","Data":"45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab"}
Jul 07 03:23:55.908551 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:55.908042    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jjq7/must-gather-6k7n4" event={"ID":"26330820-72c0-4119-8d47-3164fd6ed91f","Type":"ContainerStarted","Data":"7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f"}
Jul 07 03:23:55.924150 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:23:55.924100    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9jjq7/must-gather-6k7n4" podStartSLOduration=1.7084456399999999 podStartE2EDuration="7.924085763s" totalImagesPullingTime="6.215640123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:23:55.923369182 +0000 UTC m=+3971.330958868" watchObservedRunningTime="2026-07-07 03:23:55.924085763 +0000 UTC m=+3971.331675450"
Jul 07 03:24:03.901931 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:03.901888    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:05.320196 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:05.320166    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:06.731885 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:06.731859    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:08.141902 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:08.141875    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:09.540218 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:09.540185    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:10.920688 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:10.920661    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:12.358686 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:12.358655    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:13.783259 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:13.783225    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:15.241520 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:15.241487    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:16.718144 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:16.718121    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:18.117899 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:18.117862    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:19.513414 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:19.513381    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:20.891891 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:20.891861    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:22.280157 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:22.280124    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:23.654868 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:23.654838    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:25.066594 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:25.066566    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-cbftb_b14d1ff1-3a46-4631-b1cc-9a73657d5408/storage-initializer/0.log"
Jul 07 03:24:26.777665 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:26.777634    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-996f5cb75-q2sjq_e897bc4a-e67a-4a06-bb96-64b781e167c7/router/0.log"
Jul 07 03:24:27.973485 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:27.973448    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-996f5cb75-q2sjq_e897bc4a-e67a-4a06-bb96-64b781e167c7/router/0.log"
Jul 07 03:24:28.921248 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:28.921205    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-shqc4_d37031cb-6a87-4a89-96e7-3280ac6be8ee/manager/0.log"
Jul 07 03:24:28.932713 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:28.932690    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-2qhqj_f0f0bd37-9814-48fb-8f9e-76480fc16649/manager/0.log"
Jul 07 03:24:30.094169 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:30.094106    2570 generic.go:356] "Generic (PLEG): container finished" podID="26330820-72c0-4119-8d47-3164fd6ed91f" containerID="7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f" exitCode=0
Jul 07 03:24:30.094169 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:30.094163    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9jjq7/must-gather-6k7n4" event={"ID":"26330820-72c0-4119-8d47-3164fd6ed91f","Type":"ContainerDied","Data":"7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f"}
Jul 07 03:24:30.094616 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:30.094527    2570 scope.go:117] "RemoveContainer" containerID="7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f"
Jul 07 03:24:30.863687 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:30.863662    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9jjq7_must-gather-6k7n4_26330820-72c0-4119-8d47-3164fd6ed91f/gather/0.log"
Jul 07 03:24:34.462024 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:34.461947    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pg4n4_818a88fe-5429-40b4-998f-cd34b2a149d8/global-pull-secret-syncer/0.log"
Jul 07 03:24:34.573279 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:34.573249    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q27h5_4fd2acb5-01d8-40e2-9ffc-729199b598d0/konnectivity-agent/0.log"
Jul 07 03:24:34.687095 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:34.687068    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-229.ec2.internal_dd0c9e68fa1554fd01092c3bef64ba0f/haproxy/0.log"
Jul 07 03:24:36.478389 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:36.478355    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-9jjq7/must-gather-6k7n4"]
Jul 07 03:24:36.478819 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:36.478533    2570 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-9jjq7/must-gather-6k7n4" podUID="26330820-72c0-4119-8d47-3164fd6ed91f" containerName="copy" containerID="cri-o://45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab" gracePeriod=2
Jul 07 03:24:36.485181 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:24:36.485134    2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"26330820-72c0-4119-8d47-3164fd6ed91f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-9jjq7\"/\"must-gather-6k7n4\": pods \"must-gather-6k7n4\" not found" pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:24:36.485781 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:36.485756    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-9jjq7/must-gather-6k7n4"]
Jul 07 03:24:36.715401 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:36.715378    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9jjq7_must-gather-6k7n4_26330820-72c0-4119-8d47-3164fd6ed91f/copy/0.log"
Jul 07 03:24:36.717870 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:36.717852    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:24:36.719644 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:24:36.719619    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6k7n4\" is forbidden: User \"system:node:ip-10-0-141-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-9jjq7\": no relationship found between node 'ip-10-0-141-229.ec2.internal' and this object" podUID="26330820-72c0-4119-8d47-3164fd6ed91f" pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:24:36.778035 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:36.778016    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/26330820-72c0-4119-8d47-3164fd6ed91f-must-gather-output\") pod \"26330820-72c0-4119-8d47-3164fd6ed91f\" (UID: \"26330820-72c0-4119-8d47-3164fd6ed91f\") "
Jul 07 03:24:36.778149 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:36.778056    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf5p7\" (UniqueName: \"kubernetes.io/projected/26330820-72c0-4119-8d47-3164fd6ed91f-kube-api-access-gf5p7\") pod \"26330820-72c0-4119-8d47-3164fd6ed91f\" (UID: \"26330820-72c0-4119-8d47-3164fd6ed91f\") "
Jul 07 03:24:36.780329 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:36.780299    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26330820-72c0-4119-8d47-3164fd6ed91f-kube-api-access-gf5p7" (OuterVolumeSpecName: "kube-api-access-gf5p7") pod "26330820-72c0-4119-8d47-3164fd6ed91f" (UID: "26330820-72c0-4119-8d47-3164fd6ed91f"). InnerVolumeSpecName "kube-api-access-gf5p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:24:36.784622 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:36.784598    2570 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26330820-72c0-4119-8d47-3164fd6ed91f-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "26330820-72c0-4119-8d47-3164fd6ed91f" (UID: "26330820-72c0-4119-8d47-3164fd6ed91f"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:24:36.879358 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:36.879332    2570 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/26330820-72c0-4119-8d47-3164fd6ed91f-must-gather-output\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:24:36.879446 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:36.879372    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf5p7\" (UniqueName: \"kubernetes.io/projected/26330820-72c0-4119-8d47-3164fd6ed91f-kube-api-access-gf5p7\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 07 03:24:37.086496 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:37.086432    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26330820-72c0-4119-8d47-3164fd6ed91f" path="/var/lib/kubelet/pods/26330820-72c0-4119-8d47-3164fd6ed91f/volumes"
Jul 07 03:24:37.127172 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:37.127153    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-9jjq7_must-gather-6k7n4_26330820-72c0-4119-8d47-3164fd6ed91f/copy/0.log"
Jul 07 03:24:37.127541 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:37.127518    2570 generic.go:356] "Generic (PLEG): container finished" podID="26330820-72c0-4119-8d47-3164fd6ed91f" containerID="45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab" exitCode=143
Jul 07 03:24:37.127651 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:37.127560    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9jjq7/must-gather-6k7n4"
Jul 07 03:24:37.127651 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:37.127577    2570 scope.go:117] "RemoveContainer" containerID="45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab"
Jul 07 03:24:37.135585 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:37.135568    2570 scope.go:117] "RemoveContainer" containerID="7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f"
Jul 07 03:24:37.149317 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:37.149299    2570 scope.go:117] "RemoveContainer" containerID="45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab"
Jul 07 03:24:37.149583 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:24:37.149565    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab\": container with ID starting with 45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab not found: ID does not exist" containerID="45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab"
Jul 07 03:24:37.149656 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:37.149589    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab"} err="failed to get container status \"45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab\": rpc error: code = NotFound desc = could not find container \"45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab\": container with ID starting with 45801b15d87f0f26d694dffa859b9e1dddba975541db14b51ad1f28be3666aab not found: ID does not exist"
Jul 07 03:24:37.149656 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:37.149619    2570 scope.go:117] "RemoveContainer" containerID="7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f"
Jul 07 03:24:37.149885 ip-10-0-141-229 kubenswrapper[2570]: E0707 03:24:37.149863    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f\": container with ID starting with 7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f not found: ID does not exist" containerID="7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f"
Jul 07 03:24:37.149931 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:37.149895    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f"} err="failed to get container status \"7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f\": rpc error: code = NotFound desc = could not find container \"7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f\": container with ID starting with 7628765dbc2f087c8c5dc6704ac1893e86c6736d1bb9a852a06b888774a41d1f not found: ID does not exist"
Jul 07 03:24:39.693967 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:39.693917    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-shqc4_d37031cb-6a87-4a89-96e7-3280ac6be8ee/manager/0.log"
Jul 07 03:24:39.739245 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:39.739221    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-2qhqj_f0f0bd37-9814-48fb-8f9e-76480fc16649/manager/0.log"
Jul 07 03:24:41.271425 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.271397    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b570cce1-9eea-4a4a-8c93-54afa0e65736/alertmanager/0.log"
Jul 07 03:24:41.291989 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.291959    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b570cce1-9eea-4a4a-8c93-54afa0e65736/config-reloader/0.log"
Jul 07 03:24:41.310569 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.310546    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b570cce1-9eea-4a4a-8c93-54afa0e65736/kube-rbac-proxy-web/0.log"
Jul 07 03:24:41.341700 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.341680    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b570cce1-9eea-4a4a-8c93-54afa0e65736/kube-rbac-proxy/0.log"
Jul 07 03:24:41.361339 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.361309    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b570cce1-9eea-4a4a-8c93-54afa0e65736/kube-rbac-proxy-metric/0.log"
Jul 07 03:24:41.383360 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.383325    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b570cce1-9eea-4a4a-8c93-54afa0e65736/prom-label-proxy/0.log"
Jul 07 03:24:41.401739 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.401719    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b570cce1-9eea-4a4a-8c93-54afa0e65736/init-config-reloader/0.log"
Jul 07 03:24:41.453479 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.453426    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-t678z_a15d589e-a296-4ba9-83d9-2a8a74f8823e/cluster-monitoring-operator/0.log"
Jul 07 03:24:41.540890 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.540862    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d85c655c5-kgzgq_a22b133c-7d55-4adc-9344-49eade940c2e/metrics-server/0.log"
Jul 07 03:24:41.602773 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.602757    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6j2dr_472fd877-0e2d-4edb-a37e-5e61acfff687/node-exporter/0.log"
Jul 07 03:24:41.624853 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.624832    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6j2dr_472fd877-0e2d-4edb-a37e-5e61acfff687/kube-rbac-proxy/0.log"
Jul 07 03:24:41.645454 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.645437    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6j2dr_472fd877-0e2d-4edb-a37e-5e61acfff687/init-textfile/0.log"
Jul 07 03:24:41.927113 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.927085    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661c1aad-7c80-40b1-9938-3564b19025da/prometheus/0.log"
Jul 07 03:24:41.940838 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.940817    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661c1aad-7c80-40b1-9938-3564b19025da/config-reloader/0.log"
Jul 07 03:24:41.966096 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.966075    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661c1aad-7c80-40b1-9938-3564b19025da/thanos-sidecar/0.log"
Jul 07 03:24:41.985744 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:41.985729    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661c1aad-7c80-40b1-9938-3564b19025da/kube-rbac-proxy-web/0.log"
Jul 07 03:24:42.005640 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:42.005621    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661c1aad-7c80-40b1-9938-3564b19025da/kube-rbac-proxy/0.log"
Jul 07 03:24:42.034850 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:42.034829    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661c1aad-7c80-40b1-9938-3564b19025da/kube-rbac-proxy-thanos/0.log"
Jul 07 03:24:42.052755 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:42.052735    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661c1aad-7c80-40b1-9938-3564b19025da/init-config-reloader/0.log"
Jul 07 03:24:42.281192 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:42.281163    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d4ff86-ln78g_564204fd-ca9d-43c4-b1d9-0c546f8202e8/thanos-query/0.log"
Jul 07 03:24:42.303831 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:42.303815    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d4ff86-ln78g_564204fd-ca9d-43c4-b1d9-0c546f8202e8/kube-rbac-proxy-web/0.log"
Jul 07 03:24:42.323627 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:42.323606    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d4ff86-ln78g_564204fd-ca9d-43c4-b1d9-0c546f8202e8/kube-rbac-proxy/0.log"
Jul 07 03:24:42.345655 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:42.345634    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d4ff86-ln78g_564204fd-ca9d-43c4-b1d9-0c546f8202e8/prom-label-proxy/0.log"
Jul 07 03:24:42.370505 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:42.370485    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d4ff86-ln78g_564204fd-ca9d-43c4-b1d9-0c546f8202e8/kube-rbac-proxy-rules/0.log"
Jul 07 03:24:42.389537 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:42.389517    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d4ff86-ln78g_564204fd-ca9d-43c4-b1d9-0c546f8202e8/kube-rbac-proxy-metrics/0.log"
Jul 07 03:24:43.713196 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:43.713168    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-ljlsp_34317344-6095-46da-b8ee-00d49db5a486/networking-console-plugin/0.log"
Jul 07 03:24:44.937460 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.937431    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"]
Jul 07 03:24:44.937837 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.937825    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26330820-72c0-4119-8d47-3164fd6ed91f" containerName="copy"
Jul 07 03:24:44.937837 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.937836    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="26330820-72c0-4119-8d47-3164fd6ed91f" containerName="copy"
Jul 07 03:24:44.937914 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.937846    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26330820-72c0-4119-8d47-3164fd6ed91f" containerName="gather"
Jul 07 03:24:44.937914 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.937851    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="26330820-72c0-4119-8d47-3164fd6ed91f" containerName="gather"
Jul 07 03:24:44.937914 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.937908    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="26330820-72c0-4119-8d47-3164fd6ed91f" containerName="gather"
Jul 07 03:24:44.938007 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.937917    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="26330820-72c0-4119-8d47-3164fd6ed91f" containerName="copy"
Jul 07 03:24:44.941266 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.941247    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:44.943630 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.943611    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8t88\"/\"kube-root-ca.crt\""
Jul 07 03:24:44.943753 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.943611    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8t88\"/\"openshift-service-ca.crt\""
Jul 07 03:24:44.943753 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.943616    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8t88\"/\"default-dockercfg-s74vw\""
Jul 07 03:24:44.947827 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:44.947801    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"]
Jul 07 03:24:45.044422 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.044395    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-sys\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.044539 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.044450    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-podres\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.044539 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.044478    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nd9b\" (UniqueName: \"kubernetes.io/projected/9f0a117d-b12e-4325-841a-02a94763775d-kube-api-access-4nd9b\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.044539 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.044502    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-lib-modules\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.044539 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.044518    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-proc\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.145562 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.145539    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-podres\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.145678 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.145587    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nd9b\" (UniqueName: \"kubernetes.io/projected/9f0a117d-b12e-4325-841a-02a94763775d-kube-api-access-4nd9b\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.145776 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.145749    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-podres\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.145834 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.145762    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-lib-modules\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.145834 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.145823    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-proc\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.145945 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.145871    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-sys\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.145945 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.145867    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-lib-modules\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.145945 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.145891    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-proc\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.145945 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.145911    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f0a117d-b12e-4325-841a-02a94763775d-sys\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.152531 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.152511    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8t88\"/\"kube-root-ca.crt\""
Jul 07 03:24:45.163237 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.163220    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8t88\"/\"openshift-service-ca.crt\""
Jul 07 03:24:45.174414 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.174395    2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nd9b\" (UniqueName: \"kubernetes.io/projected/9f0a117d-b12e-4325-841a-02a94763775d-kube-api-access-4nd9b\") pod \"perf-node-gather-daemonset-jvxcn\" (UID: \"9f0a117d-b12e-4325-841a-02a94763775d\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.240108 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.240053    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-jlf4s_a482dca7-e069-4636-aefe-bf75e2d4ed17/volume-data-source-validator/0.log"
Jul 07 03:24:45.254423 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.254398    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8t88\"/\"default-dockercfg-s74vw\""
Jul 07 03:24:45.262288 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.262270    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:45.391302 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:45.391272    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"]
Jul 07 03:24:45.393767 ip-10-0-141-229 kubenswrapper[2570]: W0707 03:24:45.393738    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9f0a117d_b12e_4325_841a_02a94763775d.slice/crio-19e0e40dee7535322518c9218b696fa7c16cb03ae72fbc278833e21d679b1cd6 WatchSource:0}: Error finding container 19e0e40dee7535322518c9218b696fa7c16cb03ae72fbc278833e21d679b1cd6: Status 404 returned error can't find the container with id 19e0e40dee7535322518c9218b696fa7c16cb03ae72fbc278833e21d679b1cd6
Jul 07 03:24:46.174339 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:46.174304    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn" event={"ID":"9f0a117d-b12e-4325-841a-02a94763775d","Type":"ContainerStarted","Data":"2ad1988fc77cfbad3ceb72508bd4661bc4fe2ef1213f72f4fee89141f473fef1"}
Jul 07 03:24:46.174881 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:46.174385    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn" event={"ID":"9f0a117d-b12e-4325-841a-02a94763775d","Type":"ContainerStarted","Data":"19e0e40dee7535322518c9218b696fa7c16cb03ae72fbc278833e21d679b1cd6"}
Jul 07 03:24:46.174881 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:46.174449    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:46.190607 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:46.190554    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn" podStartSLOduration=2.190541976 podStartE2EDuration="2.190541976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 03:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:24:46.188966654 +0000 UTC m=+4021.596556342" watchObservedRunningTime="2026-07-07 03:24:46.190541976 +0000 UTC m=+4021.598131675"
Jul 07 03:24:46.292510 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:46.292486    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hfblg_17e64b2d-eebf-40dc-a342-909eb26792ba/dns/0.log"
Jul 07 03:24:46.309866 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:46.309850    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hfblg_17e64b2d-eebf-40dc-a342-909eb26792ba/kube-rbac-proxy/0.log"
Jul 07 03:24:46.332425 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:46.332406    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h4v89_785f14dc-4626-4437-be21-2761e326fe7d/dns-node-resolver/0.log"
Jul 07 03:24:46.945538 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:46.945504    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x54gh_c7c7db16-1364-477c-8b49-489562b8683c/node-ca/0.log"
Jul 07 03:24:48.154189 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:48.154161    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-996f5cb75-q2sjq_e897bc4a-e67a-4a06-bb96-64b781e167c7/router/0.log"
Jul 07 03:24:48.705509 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:48.705477    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rdp5n_d9982460-1009-4f83-8cfd-165fe6308074/serve-healthcheck-canary/0.log"
Jul 07 03:24:49.132485 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:49.132450    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-cpjkf_9f681b31-f05e-4f17-8b0d-7aaac08ea039/insights-operator/0.log"
Jul 07 03:24:49.133662 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:49.133633    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-cpjkf_9f681b31-f05e-4f17-8b0d-7aaac08ea039/insights-operator/1.log"
Jul 07 03:24:49.289730 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:49.289708    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vt8kn_edfa0e44-4c4e-4e5d-bc18-cfaf99348413/kube-rbac-proxy/0.log"
Jul 07 03:24:49.310272 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:49.310255    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vt8kn_edfa0e44-4c4e-4e5d-bc18-cfaf99348413/exporter/0.log"
Jul 07 03:24:49.329743 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:49.329722    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vt8kn_edfa0e44-4c4e-4e5d-bc18-cfaf99348413/extractor/0.log"
Jul 07 03:24:52.033065 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:52.033031    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d5cdd5cf7-9fjw7_90e7faab-e405-4dd6-a39c-dac8764beaf8/manager/0.log"
Jul 07 03:24:52.189185 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:52.189159    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-jvxcn"
Jul 07 03:24:52.745096 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:52.745072    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-ngwbc_42ca53f9-9f2d-4c40-a340-dcf857dc3f00/server/0.log"
Jul 07 03:24:53.628836 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:53.628806    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-m82hl_ea806b72-e957-42cd-9348-35c377e6c42a/manager/0.log"
Jul 07 03:24:53.672027 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:53.672007    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-g9h6p_9e242789-fd76-4af8-967f-6de77e92d596/s3-init/0.log"
Jul 07 03:24:53.688437 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:53.688419    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-g9h6p_9e242789-fd76-4af8-967f-6de77e92d596/download-hf-model/0.log"
Jul 07 03:24:56.126662 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:56.126640    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-996f5cb75-q2sjq_e897bc4a-e67a-4a06-bb96-64b781e167c7/router/0.log"
Jul 07 03:24:57.488543 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:57.488517    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-996f5cb75-q2sjq_e897bc4a-e67a-4a06-bb96-64b781e167c7/router/0.log"
Jul 07 03:24:58.796557 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:58.796530    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-bdrzv_d29f7fe1-2c3b-432f-a5a1-868515bad25c/migrator/0.log"
Jul 07 03:24:58.816448 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:58.816420    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-bdrzv_d29f7fe1-2c3b-432f-a5a1-868515bad25c/graceful-termination/0.log"
Jul 07 03:24:59.157909 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:59.157817    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-jz256_6b3022fc-1859-4338-bb06-cef488979501/kube-storage-version-migrator-operator/1.log"
Jul 07 03:24:59.159568 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:24:59.159546    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-jz256_6b3022fc-1859-4338-bb06-cef488979501/kube-storage-version-migrator-operator/0.log"
Jul 07 03:25:00.338169 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:00.338142    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sk7b9_2b740e4f-f613-4235-8aa1-ec671ed8208a/kube-multus-additional-cni-plugins/0.log"
Jul 07 03:25:00.360396 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:00.360368    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sk7b9_2b740e4f-f613-4235-8aa1-ec671ed8208a/egress-router-binary-copy/0.log"
Jul 07 03:25:00.381442 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:00.381407    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sk7b9_2b740e4f-f613-4235-8aa1-ec671ed8208a/cni-plugins/0.log"
Jul 07 03:25:00.402644 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:00.402614    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sk7b9_2b740e4f-f613-4235-8aa1-ec671ed8208a/bond-cni-plugin/0.log"
Jul 07 03:25:00.421668 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:00.421653    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sk7b9_2b740e4f-f613-4235-8aa1-ec671ed8208a/routeoverride-cni/0.log"
Jul 07 03:25:00.439651 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:00.439626    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sk7b9_2b740e4f-f613-4235-8aa1-ec671ed8208a/whereabouts-cni-bincopy/0.log"
Jul 07 03:25:00.457677 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:00.457655    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sk7b9_2b740e4f-f613-4235-8aa1-ec671ed8208a/whereabouts-cni/0.log"
Jul 07 03:25:00.540911 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:00.540886    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vdkb4_549d5421-b20a-4288-88c3-c3629f566982/kube-multus/0.log"
Jul 07 03:25:00.639081 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:00.639017    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wcsrt_e49d21df-0b01-4845-ad69-b2ad83b93187/network-metrics-daemon/0.log"
Jul 07 03:25:00.654503 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:00.654483    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wcsrt_e49d21df-0b01-4845-ad69-b2ad83b93187/kube-rbac-proxy/0.log"
Jul 07 03:25:01.876285 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:01.876256    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcklb_68468352-f49f-497c-b807-a5e0d7e50082/ovn-controller/0.log"
Jul 07 03:25:01.890033 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:01.890010    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcklb_68468352-f49f-497c-b807-a5e0d7e50082/ovn-acl-logging/0.log"
Jul 07 03:25:01.918237 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:01.918211    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcklb_68468352-f49f-497c-b807-a5e0d7e50082/kube-rbac-proxy-node/0.log"
Jul 07 03:25:01.941870 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:01.941846    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcklb_68468352-f49f-497c-b807-a5e0d7e50082/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 03:25:01.956275 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:01.956245    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcklb_68468352-f49f-497c-b807-a5e0d7e50082/northd/0.log"
Jul 07 03:25:01.974280 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:01.974253    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcklb_68468352-f49f-497c-b807-a5e0d7e50082/nbdb/0.log"
Jul 07 03:25:01.993555 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:01.993535    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcklb_68468352-f49f-497c-b807-a5e0d7e50082/sbdb/0.log"
Jul 07 03:25:02.076931 ip-10-0-141-229 kubenswrapper[2570]: I0707 03:25:02.076908    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcklb_68468352-f49f-497c-b807-a5e0d7e50082/ovnkube-controller/0.log"